
Edwin M. Lee, Mayor 



VOTING SYSTEMS 
TASK FORCE 



311 



3 
4 
5 
6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 
18 
19 



Recommendations on Voting Systems for 
the City and County of San Francisco 

A Report by the San Francisco Voting Systems Task Force (VSTF) 



June 2011 



Page 1 of 57 



i Table of Contents 



2 Section 1: Introduction and Background 5 

3 1.1 Mission and Context of the Voting Systems Task Force 

4 (VSTF) 5 

5 1.2 Background on San Francisco's Current Voting System 7 

6 1 .3 Opportunities Presented by "Next Generation" Voting 

7 Systems 8 

8 Section 2: Recommendations 10 

9 2.1. Election Records and Post-Election Audit Procedures 10 

10 2.1.1 Introduction 10 

11 2.1.2 Concepts and Definitions 10 

12 2.13 Findings 12 

13 2.1.3.1 Voting System Vulnerabilities 12 

14 2.1.3.2 Current Auditing Procedures 13 

15 2.13.2.1 Auditing Procedures for Non-RCV Contests 13 

16 2.13.2.2 Auditing Procedures for RCV Contests 13 

17 2.1.3.3 Alternative Auditing Procedures 14 

1 8 2.13.3.1 Independent Verification 14 

19 2.13.3.2 Risk-limiting Audits 15 

20 2.13.33 Ballot-level Auditing 15 

21 2.13.3.4 Assembly Bill 2023 16 

22 2.13.3.5 File Formats 16 

23 2.13.3.6 End-to-end Verification 17 

24 2.1.4 Recommendations 18 

25 2.1.4.1 Near-term Recommendations 18 

26 2.1.4.2 Longer-term Recommendations 19 

27 2.2 Balloting Systems & Services 21 

28 2.2.1 Introduction 21 

29 2.2.2 Concepts and Definitions 22 

30 2.23 Findings 23 

3 1 2.2.3.1 Ballot Accessibility and Availability 23 

32 2.23.1.1 DRE Devices for Special-Needs Voter Ballot Casting 24 

33 2.23.1.2 Physical Durability of VVPAT Ballots 25 

34 2.23.13 Other Supportive Rationale 25 

35 2.2.3.2 Ballot Marking and Casting 26 

36 2.23.2.1 Other Supportive Rationale 27 

37 2 .2 .4 Recommendations 28 

38 2.2.4.1 Ballot Accessibility and Availability 28 

39 2.2.4.1.1 Near-Term Recommendations 28 

40 2.2.4.1.2 Longer-Term Recommendations 28 



Page 2 of 57 



1 2.2.4.2 Ballot Marking and Casting 29 

2 2.2.4.2.1 Longer-Term Recommendations 29 

3 2.3 Security 30 

4 23.1 Introduction 30 

5 23.2 Concepts and Definitions 30 

6 233 Findings 31 

7 2.3.3.1 San Francisco's Current Voting System: Existing Security Issues and 

8 Mitigation 32 

9 2.33.1.1 San Francisco's Procurement Action and Voting System Security 

10 Concerns 32 

1 1 2.33.1.2 Top-to-Bottom Review 32 

12 23.3.1.2.1 Data Integrity Flaws 33 

13 23.3.1.2.2 Cryptography Problems 33 

14 23.3.1.23 Access Control and Security Mechanisms Issues 33 

15 2 3 .3 .1 .2 .4 Software Engineering Weaknesses 34 

16 2.3.3.1.2.5 "Red Team" Security Testing 34 

17 2.33.13 Security Mitigations Measures Required for Use of the Sequoia Voting 

18 System 35 

19 2 .3 .3 .2 C urrent Voting System Security Posture 35 

20 2.3.3.3 Security for San Francisco's Future Voting Systems 36 

21 2.33.3.1 Comprehensive Voting System Security Examination Not Attempted by 

22 VSTF 36 

23 2 3 .4 Recommendations 37 

24 2.3.4.1 Security Mitigations Measures Required for Use of the Sequoia Voting 

25 System 37 

26 2.3.4.2 Near- to Mid-term Recommendations 37 

27 2.3.4.3 The Current Voting System Security Posture 37 

28 2.3.4.4 Security for San Francisco's Future Voting Systems 38 

29 2.3.4.5 Longer-Term Recommendations 38 

30 2.4 Ranked-Choice Voting 40 

31 2.4.1 Introduction 40 

32 2.4.2 Concepts and Definitions 40 

33 2.43 Findings 41 

34 2.4.3.1 Public Understanding of RCV 41 

35 2.4.3.2 Reporting Preliminary Early Election Results 42 

36 2.4.3.3 Three-Choice Limit 43 

37 2 .4 .4 Recommendations 45 

38 2.4.4.1 Public Understanding of RCV 45 

39 2.4.4.2 Reporting Preliminary Early Election Results 45 

40 2.4.4.3 Three-Choice Limit 45 

41 2.5 Acquisition Strategies 46 

42 2.5.1 Introduction 46 

43 2.5.2. Concepts and Definitions 46 



Page 3 of 57 



1 2.53 Findings 46 

2 2.5.3.1 Regulatory/Certification Environment 46 

3 2.5.3.2 Business and Partnership Models 47 

4 2.5.3.3 Transparency, Source-Code Disclosure, Licensing, and Contingency 

5 Planning 49 

6 2.5.3.4 Innovation 50 

7 2.5.3.5 Software Best Practices 50 

8 2 J5 .4 Recommendations 51 

9 2.5.4.1 Regulatory/Certification Environment 51 

10 2 .5 .4 .2 B usiness and Partnership Models 51 

1 1 2.5.4.3 Transparency, Source Code Disclosure, Licensing, and Contingency 

12 Planning 52 

13 2.5.4.4 Innovation 52 

14 2.5.4.5 Software Best Practices 53 

15 Section 3: Appendices 54 

16 3.1 Appendix A: San Francisco's RCV Manual Tally Process ....54 
n 3 .2 Appendix B : Summary of Outreach 55 

18 Section 4: About the VSTF 56 

19 4.1 Membership of the VSTF 56 

20 4.2 Biographies of VSTF Members 56 

21 
22 



Page 4 of 57 



1 Section 1: 

2 Introduction and Background 

3 



4 1.1 Mission and Context of the Voting Systems Task 

5 Force (VSTF) 

6 In September 2008 the City and County of San Francisco 1 Board of Supervisors established the 

7 Voting Systems Task Force to make recommendations to that body about voting systems 

8 standards, design, and development. We define our work as follows: 

9 Mission: Our mission is to advise San Francisco on the development and/or acquisition of voting 

10 systems that ensure fair and accurate elections, achieve voter intent, and provide for transparency 

1 1 and public auditability of voting systems components and election data. 

12 Scope and Objective: Activities encompass voting systems and related elections issues that 

13 affect or are affected by voting systems and voting system acquisition in San Francisco. A 

14 "voting system" for this report is defined to be a system of hardware, software, and processes 

15 that prepares a ballot and records, collects, transmits, counts, and reports on votes and election 

16 results as cast by voters. Included in this definition are the associated reports and audit logs that 

17 provide information about management of election data in the system and system use, integrity, 

18 administrative access, configuration and configuration changes as well as documentation for 

19 support, use and training on use of the system. 

20 Our report contains recommendations, coupled with supporting rationale, for each of the five 

21 areas identified by Section 5.405(b) of the Administrative Code: 

22 1 . Standards and guidelines to gauge the adequacy, accuracy and trustworthiness of any 

23 voting system to be developed or acquired and the adequacy of any vendor or other entity 

24 that might develop and deliver such a system; 

25 2. Methods for generating or acquiring designs for a voting system that meets applicable 

26 Federal, State, County and City laws, regulations and other requirements and all other 

27 goals for the voting system while minimizing system life-cycle costs; 

28 3. Models for development of a voting system including proprietary, disclosed and open 

29 source software and hardware approaches and which address aforementioned voting 

30 systems requirements and assure a cost effective, highly reliable, maintainable system; 



'Unless specified individually as "the City" or "the County," San Francisco" will be used to refer to the 

City and County of San Francisco throughout the remainder of this report. 

2 San Francisco, Calif, Ordinance 268-08 (2008); 

www.sfbos.org/ftp/uploadedfiles/bdsupvrs/ordinances08/o0268-08.pdf . 

3 San Francisco, Calif, Ordinance 268-08 (2008), page 3, line 4; 

www.sfbos.org/ftp/uploadedfiles/bdsupvrs/ordinances08/o0268-08.pdf. 
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1 4. Business models, including the City and County of San Francisco acting as its own 

2 vendor, which promote the transparency of all aspects of design, development, 

3 production and the business relationship of all parties associated with production, 

4 delivery, implementation and use of the voting system; 

5 5. Any other considerations related to voting systems that will promote public trust in the 

6 conduct and results of elections. 

7 Recommendations are presented in five topic areas: election records and post-election audit 

8 procedures; balloting systems and services; security; Ranked-Choice Voting; and acquisition 

9 strategies. 

10 This report is not intended to be a complete statement of requirements or technical specifications, 

1 1 and it is not an exhaustive study of all topics related to voting systems. Rather, it provides a 

12 framework that will guide San Francisco as it seeks its next voting system. A coordinated effort 

13 will be needed to establish a policy direction for San Francisco and to support its Department of 

14 Elections through a robust and forward- thinking process. We recognize that the City may wish to 

15 initiate further investigation of certain topic areas as it considers a direction for San Francisco's 

16 next voting system. We recommend that the San Francisco Board of Supervisors establish a body 

17 or process to implement the recommendations contained in this report and to maintain steady 

18 focus on this issue over coming months and years. 

19 Time Frame for Recommendations: San Francisco is currently under contract with Sequoia 

20 Voting Systems (which was acquired by Dominion Voting Systems Corporation in 20 10), 4 and 

21 has the option to extend that contract through elections in 2013. We have identified several 

22 opportunities for improving public confidence in the City's use of Sequoia Voting Systems. 

23 However, this report primarily offers guidance regarding San Francisco's next voting system. 

24 Our recommendations can be found in Section 2 of this report. 

25 Audiences: Our recommendations are intended to provide guidance to a variety of audiences 

26 including the following: 

27 • the San Francisco Board of Supervisors 

28 • the San Francisco Department of Elections 

29 • the San Francisco Elections Commission 

30 • San Francisco voters 

31 



4 Dominion Voting Systems Corporation website (2010), "Dominion Voting Systems Corporation 
Acquires Assets of Sequoia Voting Systems," (news release); 
http://www.dominionvoting.com/images/pdfs/DominionAcquiresSequoiaFinal.pdf . 
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1 1.2 Background on San Francisco's Current Voting 

2 System 

3 On March 31, 2005, the San Francisco Department of Elections (DOE) initiated a Request for 

4 Proposals (RFP) process seeking bids for a new voting system (including equipment and 

5 services) to collect, count, tabulate, and report votes. 5 In December 2007, the San Francisco 

6 Board of Supervisors approved a contract with Sequoia Voting Systems for voting 

7 systems/services. 6 Sequoia replaced Elections Systems and Software (ES&S) with which the 

8 City had been under contract through the 2007 election cycles. 

9 The Sequoia system was implemented beginning with the February 2008 election. The contract 

10 runs through December 201 1 . The contract with Sequoia Voting Systems for a voting system and 

1 1 associated services is valued at $12,650,233.35 (per Resolution 654-07). 7 The DOE has the 

12 option to renew the contract two times, each time for one year and has indicated that it 

13 anticipates extending the Sequoia contract through the end of 2013. Were it to do so, the DOE 

14 estimates that annual maintenance would be approximately $400,000, and services per election 

15 would be approximately $500,000. With three elections scheduled in 2012, the projected cost 

16 would be approximately $1.9 million. With one election scheduled in 2013, the projected cost 

17 would be approximately $900,000 (two year total: $2.8 million). 8 

18 Sequoia Voting Systems was acquired by Dominion Voting Systems in June 2010. Subsequently, 

19 the City accepted the assignment of the contract from Sequoia to Dominion. In this report, 

20 San Francisco's voting system is referred to as the "Sequoia Voting System." 

21 San Franciscan voters use an optical scan voting machine to cast their ballots at the polling place. 

22 This machine is a paper-based voting system that optically scans the marks that voters make on a 

23 paper ballot and counts the votes electronically when the ballot is inserted. Additionally, each 

24 polling place has one Sequoia AVC Edge accessible touchscreen voting machine. 9 

25 



5 City and County of San Francisco, DOE (6 February 2007), "Contract for New Voting System" (memo 

from DOE Director John Arntz to Merrick Pascual, Budget Analyst, Board of Supervisors); 

http://www.sfgov2.org/ftp/uploadedfiles/elections/Announcements/MR2007/20070206.pdf . 

6 City and County of San Francisco government website, "City and Country of San Francisco — NFAMIS 

Blanket Purchase Order Writing"; 

http://www.sfgov2.org/Modules/ShowDocument.aspx?documentid=152 . 

7 San Francisco Board of Supervisors website, "Resolution No. 65-08," 

http://www.sfbos.org/ftp/uploadedfiles/bdsupvrs/resolutions08/r0065-08.pdf 

8 San Francisco Department of Elections, e-mail message to Jody Sanford, April 14, 2010. 

9 Nataliya Kuzina, Deputy Director, San Francisco Department of Elections, e-mail message to Jody 

Sanford, April 19,2011. 
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1 The technical specifications of San Francisco's current voting system are as follows: 

2 Optech Insight Plus 

3 APX Firmware K2. 16.080626. 1320 

4 HPX Firmware Kl. 44.08050 1.1 500 
5 

6 Optech 400-C Hardware version 3 .OOP 

7 WinETP Software version 1.16.6 
8 

9 AVC Edge Model II 
10 Firmware version 5.0.24 

11 

12 WinEDS versions 3.1.012 and 4.0.1 16B 

13 WinEDS Extended Services 1 .0.47 

14 WinEDS Election Reporting Software 4.0.44 
15 

1 6 Card Activator version 5.0.21 
17 

1 8 MemoryPack Receiver (MPR) Hardware version D 

19 Firmware version 3.01.080422.0522 

20 1.3 Opportunities Presented by "Next Generation" 

21 Voting Systems 

22 John Arntz has been the director of the San Francisco Department of Elections (DOE) since 

23 2002. Under his capable leadership, DOE has conducted well-run elections. Yet, the VSTF 

24 believes that there is room to improve the underlying voting system and the procedures that 

25 accompany the elections process. We have identified opportunities for improvement in several 

26 areas: 

27 • intent of voter and accessibility 

28 • audit and verification procedures 

29 • security 

30 • transparency 

3 1 These issues exist across the entire election systems landscape. In order to address them, there is 

32 a need for innovation in the voting systems marketplace. Yet there are many barriers that limit 

33 advancement and prevent new players from entering the market: the regulatory environment at 

34 the state and federal level is shifting and cumbersome; developing, testing, and certifying new 

35 systems are costly endeavors and time-consuming; the voting systems market is fragmented with 

36 multiple jurisdictions and differing systems requirements; and many jurisdictions lack adequate 



Nataliya Kuzina, Deputy Director, San Francisco Department of Elections, e-mail message to Jody 
Sanford, April 19,2011. 
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1 funding to invest in a new system. (The regulatory environment and acquisition marketplace is 

2 discussed in detail in Section 2.5: Acquisition Strategies.) 

3 Given these challenges, San Francisco would be prudent (a) to begin immediately considering 

4 the characteristics of the voting system it would like to implement after the contract for its 

5 current voting system terminates, and (b) to consider the acquisition model it will use to obtain a 

6 new system. In fact, the City would be on par with other jurisdictions. Across the nation, 

7 jurisdictions are grappling with how to provide elections that are accurate, fair, secure, 

8 transparent, and accessible, and with how to evaluate the merits of various systems and 

9 acquisition models. 

10 The conversation about next generation voting systems is generating opportunities for 

1 1 collaboration and information sharing. An effort to study future voting systems has been 

12 undertaken by at least two other jurisdictions, including the following: 

13 1. County of Los Angeles (California) Voting Systems Assessment Project (VSAP) 

14 http://www.lavote.net/voter/VSAP 

15 2. Travis County (Texas) Elections Study Group 2009 

16 http://www.co.travis.tx.us/county clerk/election/study group 2009 

17 We acknowledge that the obstacles to progress are significant and that jurisdictions must balance 

18 aspiration with pragmatism. However, we believe that San Francisco should be an active 

19 participant in the movement toward modernized voting systems, and it should consider a broad 

20 range of possibilities regarding the business and partnership model it will pursue to 

21 acquire/develop its next voting system. 

22 While a flawless voting system is not attainable, the VSTF members hope that this strategic 

23 guidance will help San Francisco implement a voting system that earns the highest level of 

24 public confidence. 

25 
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1 Section 2: 

2 Recommendations 

3 2.1. Election Records and Post-Election Audit 

4 Procedures 

5 2.1.1 Introduction 

6 This section concerns the records generated in the course of an election and the procedures for 

7 checking records to verify that the election was conducted properly. Comprehensive records and 

8 audit procedures are essential for ensuring a correct outcome, deterring fraud, building public 

9 confidence in elections, and understanding how to improve the election system. Though there are 

10 many types of audits, this section deals only with post-election verification of the results. 

11 2.1.2 Concepts and Definitions 

12 Election records include paper or electronic records at all stages of an election, such as the 

13 following: 

14 • Voter registrations: lists of the registered voters 

15 • Election definitions: lists of the contests and candidates in the election and which groups 

16 of voters are eligible to vote in which contest 

17 • Ballot definitions: descriptions of the contents and layout of each type of blank ballot 

18 • Cast vote records (CVRs): electronic records of the choices that a voter made 

19 • Audit logs, event logs, and error reports: timed records of events that took place during 

20 the election (e.g. accessing of sensitive information, opening or closing of polls, casting 

21 of ballots, granting or revocation of access, actions by election workers) 

22 • Canvass records: all records used to reconcile vote totals during the post-election 

23 canvass period (period between election night and the date an election is certified), 

24 including ballot reconciliation sheets, records establishing chain of custody, and other 

25 precinct records 

26 • Vote counts: counts of the votes (usually within an election district) 

27 • Election outcome: the winning candidate in a contest, or the winning side of a 

28 referendum, as determined by the vote counts from all districts 

29 • Election results: the final report of overall vote counts and outcomes, including the 

30 number of ballots cast, voter registration and turnout percentages, and other election 

3 1 statistics 

32 A post-election manual tally (sometimes called a post-election audit) is a procedure conducted 

33 after an election to check the vote counts. It is usually performed by dividing the cast ballots into 
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1 groups called audit units, selecting some fraction of the audit units for a manual count, and 

2 checking that the manual counts for each unit match the vote tallies from the election. The 

3 California Elections Code, Section 15360, 11 currently requires a manual tally of the ballots from 

4 1% of the precincts. 

5 A risk-limiting audit is an audit that ensures a high, pre-specified chance of detecting and 

6 correcting an incorrect election outcome. Any auditing procedure that can provide such a 

7 guarantee qualifies as a risk-limiting audit. For the purpose of this definition, the correct outcome 

8 is the outcome that a full hand count of all the ballots would have produced. Audits can be made 

9 risk-limiting by establishing specific criteria under which a full hand count must occur. Typically 

10 a risk- limiting audit involves hand-counting a randomly selected sample of the ballots (where the 

1 1 number of ballots to count depends on how close the contest was), comparing the hand-verified 

12 results to the vote tallies, and escalating to a full hand count if the error is sufficiently large. For 

13 example, to limit the risk of an incorrect outcome to 1%, the sampling procedure and escalation 

14 criteria must be chosen such that there is at least a 99% chance of escalating to a full hand count 

15 when the outcome is incorrect. 

16 Ranked-Choice Voting (RCV) is an election method in which each voter ranks the candidates 

17 and the votes are counted through a multiple-round elimination process. This method is also 

18 known as Instant-Runoff Voting or the Alternative Vote. As currently implemented in San 

19 Francisco, each voter indicates a first choice, an optional second choice, and an optional third 

20 choice for an elected office. In the first round of counting, all ballots are assigned to their first 

21 choices. If one candidate now has a majority of the ballots, that candidate wins. If not, the 

22 candidate with the least ballots is eliminated; ballots with that candidate as their first choice are 

23 then reallocated to their second choice, or set aside as exhausted ballots if there is no second 

24 choice. Rounds of counting and elimination repeat — always assigning each ballot to its highest- 

25 ranked non-eliminated candidate — until one candidate has a majority of the non-exhausted 

26 ballots. 

27 Election Markup Language (EML) is a suite of XML-based data formats for election records, 

28 defined by the Organization for Advancement of Structured Information Standards (OASIS). The 

29 current version is EML 5.0 and work on EML 6.0 is under way. EML defines several different 

30 data formats for different kinds of records; each format is identified by a number. 

31 



"California Legislative Counsel government website, "Election Code Section 15360"; 
http://www.leginfo.ca.gov/cgi-bin/displaycode?section=elec&group=l 500 l-16000&file=l 5360 . 
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1 2.1.3 Findings 

2 2 .1 .3 .1 Voting System Vulnerabilities 

3 Numerous independent investigations have discovered serious security weaknesses and design 

4 errors in widely used electronic voting equipment. Some examples are cited as follows: 

5 • In 2004, four computer security experts examined the source code of a DRE voting 

12 

6 machine and found it to be "far below even the most minimal security standards 

7 applicable in other contexts." 

8 • In 2006, investigators at Princeton University demonstrated that it is possible to construct 

9 a software virus that spreads from voting machine to voting machine — even when the 

10 machines are not connected to a network — while altering votes in an undetectable 

11 fashion. 13 

12 • In 2007, a team of reviewers appointed by the California Secretary of State found major 

13 security flaws in three of the major brands of voting systems used in California, 14 ' 15 ' 16 ' 17 

14 including vulnerability to infection by a software virus in some cases. 

1 8 

15 • In 2008, the election system in Humboldt County erroneously deleted 197 ballots. 

16 Voting machines are still perceived as untrustworthy in the public consciousness. The 

17 investigations mentioned above were widely publicized, and there continues to be a steady flow 

18 of news headlines raising concerns about flaws and reliability problems with voting machines. 

19 Finding 1: It is not safe to rely solely on electronic voting equipment for accurate results. 



12 Kohno, Tadayoshi, Adam Stubblefield, Aviel D. Rubin, and Dan S. Wallach (2004). Analysis of an 
Electronic Voting System. In Proceedings of the 2004 IEEE Symposium on Security and Privacy, pp. 27- 
40. IEEE Computer Society Press. 

13 Feldman, Ariel J., J. Alex Halderman, Edward W. Felten (2007). "Security Analysis of the Diebold 
AccuVote-TS Voting Machine." In Proc. 2007 USENIX/ ACCURATE Electronic Voting Technology 
Workshop (EVT/WOTE '07); http://citp.princeton.edu/pub/ts06EVT.pdf . 

14 California Secretary of State Debra Bowen government website, "California Top-to-Bottom Review of 
Voting Systems" (TTBR); http://www.sos.ca.gov/voting-systems/oversight/top-to-bottom-review.htm. 
15 Calandrino, Joseph A., Ariel J. Feldman, J. Alex Halderman, David Wagner, Harlan Yu, and William P. 
Zeller (2007a). "Source Code Review of the Diebold Voting System." (report, University of California, 
Berkeley under contract to the California Secretary of State's TTBR); http://www.sos.ca.gov/voting- 
systems/oversight/ttbr/diebold-source-public-jul29.pdf . 

"Tnguva, Srinivas, Eric Rescorla, Hovav Shacham, and Dan S. Wallach (2007). "Source Code Review of 
the Hart InterCivic Voting System." (report, University of California, Berkeley under contract to the 
California Secretary of State's TTBR); http://www.sos.ca.gov/voting-systems/oversight/ttbr/Hart-source- 
public.pdf . 

17 Blaze, Matt, Arel Cordero, Sophie Engle, Chris Karlof, Naveen Sastry, Micah Sherr, Till Stegers, and 

Ka-Ping Yee (2007). "Source Code Review of the Sequoia Voting System." (report, University of 

California, Berkeley under contract to the California Secretary of State's TTBR); 

http://www.sos.ca.gov/voting-svstems/oversight/ttbr/sequoia-source-public-iul26.pdf . 

ls California Secretary of State Debra Bowen government website, "Report to the Election Assistance 

Commission Concerning Errors and Deficiencies in Diebold/Premier GEMS Version 1.18.19," (2009). 

https://www.sos.ca.gov/voting-systems/vendors/premier/sos-humboldt-report-to-eac-03-02-09.pdf . 
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1 Finding 2: Public confidence in electronic voting has been diminished by the discoveries 

2 of serious flaws in electronic voting systems. 

3 2.132 Current Auditing Procedures 

4 San Francisco's post-election audit is known as the "1% Manual Tally," in which the ballots 

5 from a random selection of precincts are manually counted. 19 The manual counts are checked 

6 against machine reports at the precinct level. For speed and accuracy, the contests are counted 

7 one at a time; that is, each counting team counts a single contest for an entire precinct, then 

8 counts the next contest for the entire precinct, and so on. 

9 We inquired as to the procedure taken when the audit appears to be at variance with the reported 

10 election results. When there is a discrepancy of even one vote, the ballots are counted again, with 

1 1 particular attention to counting the ballots as a machine would count them, not as a human would 

12 interpret the voter's intent. That is, the audit seeks a way to interpret the ballots that confirms the 

13 machine result. If a discrepancy remains after a second count, the audit team fills out a Manual 

14 Tally Incident Report, which is reviewed by supervisors in charge of the canvass. There is no 

15 formal written procedure for escalating the audit or challenging the election results based on such 

16 a discrepancy. 

17 Finding 3: The current post-election audit procedure does not establish a known limit on 

1 8 the risk of an incorrect outcome. 

19 2.1.3.2.1 Auditing Procedures for Non-RCV Contests 

20 For a regular contest, the manual count produces a tally of the number of votes for each 

21 candidate. These numbers are then compared directly to the vote counts on the machine report 

22 for the precinct. The counting process is quite fast, because the ballots are first sorted into piles 

23 (one pile for each candidate), and then each pile is counted. We watched a video of the manual 

24 tally for a ballot measure; a member of the team counted the "Yes" pile, speaking "Yes, yes, yes, 

25 yes, yes. . ." at a rate of about two ballots per second. 

26 If this manual tally process were carried out for every precinct, it would give assurance that the 

27 counts are correct in every precinct, and thus the totals are correct for the entire election, and 

28 thus the outcomes (winners) are also correct. Performing this process for a randomly selected 

29 fraction of the precincts therefore verifies the outcome with a known level of confidence. This 

30 level of confidence can be calculated, and it depends on how many precincts are checked, the 

3 1 number of ballots, and the margin of victory. A 10% tally provides higher confidence than a 1% 

32 tally, and a 100% tally provides complete confidence. 

33 2.1.3.2.2 Auditing Procedures for RCV Contests 

34 For an RCV contest, the team manually counts the first choices, second choices, and third 

35 choices separately, as if they were three independent contests, resulting in three counts for each 

36 candidate. These are compared directly to the machine report, which also provides vote counts of 



San Francisco Department of Elections, SF RCV BDProcedures2009-Final.xls, electronic file. 
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1 each RCV contest as though it were three independent contests. Next, the team carries out the 

2 RCV elimination process at the precinct level. That is, if no candidate has a simple majority of 

3 the first-choice votes in the precinct, then the candidate with the lowest number of first-choice 

4 votes in the precinct is eliminated; those ballots are transferred to piles for their second-choice 

5 candidates, and so on. 

6 Since the actual election outcome is determined by elimination based on totals for the entire 

7 election, the sequence of candidates eliminated during the manual precinct tally bears no 

8 relationship to the actual elimination sequence. Also, checking the three independent totals does 

9 not verify the outcome because the outcome depends on which first-choice votes are cast with 

10 which second-choice votes — not just how many of each there are. Thus the RCV manual tally 

1 1 process does not verify the outcome of the election (see Appendix A for a detailed example). 

12 Finding 4: The manual tally procedure for RCV contests is significantly more complex 

13 than the procedure for non-RCV contests. 

14 Finding 5: The manual tally procedure does not verify the outcome of RCV contests. 

15 2 .1 .3 .3 Alternative Auditing Procedures 

16 2.1.3.3.1 Independent Verification 

17 The deletion of 197 ballots in Humboldt County led to the certification of incorrect results in the 

18 November 4, 2008, General Election. The discrepancy went undetected until it was discovered 

19 by an audit conducted by the Humboldt County Election Transparency Project. 20 The ballots 

20 were scanned with a general-purpose, high-speed office scanner. A pre-imprinter attached to the 

21 scanner printed a unique serial number on each ballot before scanning. The resulting scanned 

2 1 

22 images were then counted by an image analysis program called TEVS, written by Mitch 

23 Trachtenberg. TEVS is freely available under an open source license and has been developed 

24 further since 2008. 

25 Finding 6: An independent verification of an election has been successfully conducted 

26 by scanning and counting ballots using ordinary office equipment and free software, and 

27 such procedures can be effective at detecting errors in election results. 

28 



Humboldt Election Transparency Project, last accessed on June 23, 201 1; http ://humtp .com/ . 
2 'Trachtenberg Election Verification System, last accessed on June 23, 201 1; 
http://code.google.eom/p/tevs/ . 
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1 2 .1 .3 .3 .2 Risk-limiting Audits 

2 In 2008, Joseph Hall et al. conducted risk-limiting audits of four contests from elections that took 

3 place in California's Marin, Yolo, and Santa Cruz counties. The authors reported that "[t]he cost 

4 and the time required were modest. . . .There remains room for big gains in efficiency — that is, for 

5 reducing the number of ballots that must be counted to confirm an election outcome that is, in 

22 

6 fact, correct." 

7 In 2009, risk-limiting audits were performed for two contests in Yolo County, as reported by 

23 

8 Philip Stark." In one case, the audit units were batches of between 200 and 600 ballots, and the 

9 risk-limiting audit required hand-counting 1,437 ballots — a little more than 1 1% of the ballots 

10 cast. In the other case, the audit units were individual ballots, and the risk-limiting audit required 

1 1 hand-counting only 32 ballots. 

12 Finding 7: Risk-limiting audits have been carried out successfully in California. 

13 We note that several risk-limiting audit methods have been proposed and published in peer- 

14 reviewed literature. One notable example is the method proposed in "Super-Simple Simultaneous 

15 Single-Ballot Risk-Limiting Audits," 4 which audits all the contests on the ballot at once, 

16 requires just one parameter to be calculated by a formula (which needs to be calculated only once 

17 before the audit begins), and has a simple method for determining how many ballots to check. 

1 8 However, all the proposed methods so far assume a non-RCV contest: there appear to be no 

19 peer-reviewed, published methods for risk-limiting audits of RCV contests. 

20 Finding 8: There is at least one peer-reviewed risk-limiting audit method for non-RCV 

21 contests that is practical and straightforward to carry out. 

22 Finding 9: There do not appear to be any peer-reviewed risk-limiting audit methods for 

23 RCV contests that have yet been published. 

24 2.1.3.3.3 Ballot-level Auditing 

25 In addition to Stark, other researchers also report that auditing at the individual ballot level 

26 dramatically reduces the number of ballots that need to be hand-counted in order to achieve a 

25 

27 high degree of confidence. Calandrino et al. (2007b) have proposed a method of ballot-level 

28 auditing that uses a machine to mark each ballot with a unique number, so that randomly selected 



Hall, Joseph Lorenzo, Luke W. Miratrix, Philip B. Stark, Melvin Briones, Elaine Ginnold, Freddie 
Oakley, Martin Peaden, Gail Pellerin, Tom Stanionis, Tricia Webber (2009). "Implementing Risk- 
Limiting Post-Election Audits in California"; 
http://www.usenix.org/event/evtwote09/tecrVfullj3apers/hall.pdf . 

23 Stark, P.B. (2009). "Efficient Post-Election Audits Of Multiple Contests: 2009 California Tests." 
(Refereed paper presented at the 2009 Conference on Empirical Legal Studies.); 
http://ssrn.com/abstract=14433 14 . 

24 Stark, P.B. (2010). "Super-Simple Simultaneous Single-Ballot Risk-Limiting Audits," in the 2010 
Electronic Voting Technology Workshop/Workshop on Trustworthy Elections (EVT/WOTE '10); 
http://www.usenix.org/events/evtwotelO/tech/full papers/Stark.pdf . 

25 Calandrino, Joseph A., J. Alex Halderman, and Edward W. Felten (2007b). "Machine -Assisted Election 
Auditing"; http://www.usenix.org/events/evt07/tech/fullj3apers/calandrino/calandrino.pdf . 



Page 15 of 57 



1 ballots can be individually retrieved and checked against their corresponding cast vote records. 

2 Even without such markings, keeping the ballots stacked in the same order that they were 

26 

3 scanned is sufficient to make ballot-level auditing possible. The number of ballots to check 

4 depends on the margin of victory; closer contests require more manual checking. Calandrino 

5 et al. (2007b) analyzed the statewide contests in the Virginia elections in November 2006 and 

6 found that in order to achieve a 99% confidence level of detecting an incorrect outcome, a ballot- 

7 level audit would require hand-counting 40 times fewer ballots than a precinct-level audit. 

8 Finding 10: Performing a risk-limiting audit with large audit units (e.g. randomly 

9 selecting entire precincts for manual counting) is likely to be more expensive than a 1% 

10 manual tally. 

1 1 Finding 11: Performing a risk-limiting audit at the ballot level (i.e. randomly selecting 

12 individual ballots for manual counting) is likely to be considerably cheaper than a 1% 

13 manual tally, while providing stronger assurance that the outcome is correct. 

14 2.1.33.4 Assembly Bill 2023 

27 

15 We note that California Assembly Bill 2023 (AB 2023) authorizes the establishment of a 

16 groundbreaking pilot program to conduct risk-limiting audits in "5 or more voluntarily 

17 participating counties" during 201 1 . The program will yield a report to the California 

18 Legislature evaluating the effectiveness and efficiency of the audits. We find that the definition 

19 of "risk-limiting audit" given in AB 2023 matches the meaning intended in this report. 

20 Finding 12: The AB 2023 pilot program provides a valuable opportunity to conduct 

21 officially recognized risk-limiting audits, advance the state of the art in post-election 

22 auditing procedures, and achieve higher confidence in election outcomes. 

23 2.1.33.5 File Formats 

24 Those who have conducted the aforementioned audits also reported that "[a] great deal of 

25 scripting and hand editing was required to make the exported data [from Election Management 

26 Systems] useful. . . .Election auditing requires better 'data plumbing' than EMS vendors currently 

27 provide. . . .One suitable format is the OASIS Election Markup Language (EML). . . ." 



Note: In order to preserve voter anonymity, it is important not to order the ballots in a way that can be 
correlated with the order in which public records show the voters as having cast their votes. 
27 0fficial California Legislative Information, "Assembly Bill 2023"; http://www.leginfo.ca.gov/pub/09- 
10/bill/asm/ab 2001-2050/ab 2023 bill 20100719 chapteredpdf . 

28 Hall, Joseph Lorenzo, Luke W. Miratrix, Philip B. Stark, Melvin Briones, Elaine Ginnold, Freddie 
Oakley, Martin Peaden, Gail Pellerin, Tom Stanionis, Tricia Webber (2009). "Implementing Risk- 
Limiting Post-Election Audits in California"; 
http://www.usenix.org/event/evtwote09/tech/fullj3apers/hall.pdf . 
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1 Neal McBurnett worked with the Boulder County Elections Division to conduct an audit for the 

2 2008 General Election in Boulder County, Colorado, and reported the following: 

3 • Most of the reports produced by the Hart tally system were poorly specified or hard to 

4 parse for auditing. 

5 • The Hart tally system produced an XML report that was usable for auditing, though it 

6 still lacked some important information and did not adhere to the EML standard. 

7 • Effective audits are easier and require less hand counting to achieve a similar level of 

8 confidence if results are reported in smaller audit units. 

9 Both of these reports point to non-proprietary reporting formats, specifically EML. We are also 

10 aware of IEEE P-1622, which is another voting data standard under development with more of a 

1 1 focus on elections in the United States. We have not reviewed the specification for P-1622, as the 

12 IEEE P-1622 Working Group's working documents are not freely available on its website. If and 

13 when P-1622 is a fully developed, freely available open standard with comparable 

14 expressiveness to EML, it may also be a suitable option. 

15 Finding 13: The use of proprietary, vendor-specific data formats increases the difficulty 

16 of conducting an audit or forensic investigation. 

17 Finding 14: Election Markup Language is a suitable structured data format for enabling 

1 8 efficient post-election audits. 

19 2.1.3.3.6 End-to-end Verification 

20 Another way to establish confidence in an election is to provide the voters with a way to verify 

2 1 that their own votes were correctly recorded and included in the tally. Voting systems that make 

22 this possible are said to offer end-to-end verification. At the same time, however, it is important 

23 to avoid enabling voters to prove how they voted in such a way that they can sell their votes, and 

24 also to avoid enabling voters to fraudulently claim that their votes were misrecorded. Although 

25 this is a tricky set of requirements to satisfy all at once, there is a substantial body of research 

26 and invention in voting systems that actually do have all of these properties. To cite one 

27 example, a system called Scantegrity II 30 allows voters to note confirmation codes for their 

28 choices and check those codes against a published list of the codes for all the cast ballots, but it 

29 does not allow them to prove to others which candidates those codes represented. The City of 

30 Takoma Park, Maryland, used Scantegrity II for its November 2009 election. 

31 Finding 15: There is at least one voting system offering paper-based end-to-end 

32 verification that has been used to conduct a real election. 



29 McBurnett, Neal (2008). "Obtaining Batch Reports for Audits from Election Management Systems: 
Election Audits and the Boulder 2008 Election" National Institute of Standards and Technology; 
http://www.nist.gov/itl/vote/upload/neal-mcburnett-boulder-paper.pdf . 
30 Scantegrity, last accessed on June 23, 201 1; http://www.scantegrity.org/ . 
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1 2.1.4 Recommendations 

2 Based on the findings above, the VSTF makes the following recommendations. 

3 Recommendations 1 through 7 can begin implementation now. Recommendations 8 through 12 

4 concern longer-term or more speculative changes, such as the criteria for San Francisco's next 

5 voting system. Below, the phrase "EML or an equivalent open standard" refers to a publicly 

6 available, freely licensed format of equivalent expressiveness to EML, established by a vendor- 

7 independent national or international technical standards body. 

8 2.1.4.1 Near-term Recommendations 

9 1 . Publish all election records on the city's website, redacting records only as necessary to 

10 protect the anonymity of each voter's votes and the privacy of each voter's personally 

1 1 identifying information. Give public notice when records are published. Whenever 

12 feasible, use EML or an equivalent open standard format for the published records. The 

13 VSTF recommends prioritizing these four types of records first: 

14 a. Tallies of the vote counts, under-votes, and over-votes from each precinct: 

15 Publish (using EML section 500 or equivalent formats) as soon as possible after 

16 each precinct closes its polls. For RCV contests, publish the tallies of each 

17 preference level, to the extent that these tallies can be compared against totals 

1 8 published at the polling place in order to verify the correct transfer of ballots to 

19 the central election office. 

20 b. Text files of cast vote records, which are currently called "ballot image files ": 

21 For precinct-scanned ballots, publish as soon as the memory packs are loaded; for 

22 centrally scanned ballots, publish as soon as the ballots are centrally scanned. 

23 These must be published before any precincts are randomly selected for audits. 

24 c. Election definitions: Publish (using EML section 200 and 600 or equivalent 

25 formats) as soon as the Qualified Candidate List and Official Measures List are 

26 complete. 

27 d. Ballot definition files: Publish (in the current proprietary format) as soon as ballot 

28 layouts are complete. When EML or an equivalent open standard format is used 

29 (see Recommendation 7), publish in that format. 

30 2. Correct the audit procedure for RCV contests in such a way that a 100% tally would 

3 1 actually ascertain the outcome. In particular, as recommended by the California Secretary 

32 of State, use entire-election totals — not precinct vote totals — to determine which 

33 candidates to eliminate. 31 

34 3. Pursue participation in the post-canvass risk-limiting audit pilot program authorized by 

35 California AB 2023. 



31 California Secretary of State Debra Bowen government website. Debra Bowen (2010) "Instant Runoff 
Voting in Charter Counties and Charter Cities"; http://www.sos.ca.gov/voting- 
systems/oversight/directives/irv-guidelines.pdf . 
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1 4. Define, pilot, and use a ballot-level risk-limiting audit procedure for all non-RCV 

2 contests, taking guidance from Hall et al.'s "Implementing Risk-Limiting Post-Election 

32 

3 Audits in California" and considering as one option Stark's "Super-Simple 

33 

4 Simultaneous Single-Ballot Risk-Limiting Audits." 

5 5. At such time as a peer-reviewed method for risk-limiting audits of RCV contests has been 

6 published, define, pilot, and use a ballot-level risk-limiting audit procedure for all RCV 

7 contests. 

8 6. Permit academic organizations 34 to publicly request and obtain timely access to all the 

9 paper ballots (without any information linking ballots to voter identities) for the sole 

10 purpose of digitally scanning the ballots and analyzing the scanned images to 

1 1 independently verify election results, and to publish their findings from such 

12 verification. 

13 7. Permit academic organizations to publicly request, obtain, and study machine audit logs 

14 from which any information identifying individual voters has been removed, and to 

15 publish their findings from such study. 

16 2.1.4.2 Longer-term Recommendations 

17 8. Consider broadening the audience with access in Recommendations 6 and 7 to include 

18 other organizations that serve the public interest, or all members of the public, under 

19 conditions that limit conflicts of interest, provide full transparency, protect voter privacy, 

20 and discourage vote-selling. 



Hall, Joseph Lorenzo, Luke W. Miratrix, Philip B. Stark, Melvin Briones, Elaine Ginnold, Freddie 
Oakley, Martin Peaden, Gail Pellerin, Tom Stanionis, Tricia Webber (2009). "Implementing Risk- 
Limiting Post-Election Audits in California"; 
http://www.usenix.org/event/evtwote09/tech/full papers/hall.pdf . 

33 Stark, P.B. (2010). "Super-Simple Simultaneous Single-Ballot Risk-Limiting Audits," in 2010 
Electronic Voting Technology Workshop/Workshop on Trustworthy Elections (EVT/WOTE '10); URL: 
http://www.usenix.org/events/evtwotelO/tech/full papers/Starkpdf . 

34 Note: Due to the precedent set by the Humboldt County Election Transparency Project, the 
VSTF finds great potential value in enabling independent parties to scan and analyze the ballots. 
However, we also recognize that making all ballots available to the public creates concerns about 
voter privacy and may enable large-scale vote-selling. Because we believe that independent 
verification is a powerful way to establish voter confidence, we prefer to see short-term action 
rather than have such action delayed, perhaps indefinitely, by the controversy that the public 
distribution of ballots would raise. Thus, we propose the compromise of making the ballots 
available for scanning to academic organizations, under the assumption that such organizations 
have sufficient reputations and oversight to be trusted not to engage in large-scale vote-selling 
schemes. We do not intend this recommendation to prohibit members of the public from 
collaborating with academic organizations to participate in the verification process. In fact, we 
encourage a publicly transparent process. 

35 By way of example, two such systems are or will become available from TEVSystems 
( http://www.TEVSystems.com , http://code.google.eom/p/tevs/ ) and the Clear Ballot Group 
( http://www.ClearBallot.com ). 
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1 9. Use EML or an equivalent open standard format internally within the Department of 

2 Elections as the primary data format for election definitions and results. 



3 10. Announce an acquisition preference for voting systems that facilitate auditing of 

4 individual randomly selected ballots, for example, by exporting digital cast vote records, 

5 by exporting scanned images of ballots, and/or by printing a unique identifier on each 

6 ballot at the time the ballot is scanned in order to associate each physical ballot with its 

7 digital cast vote record. 

8 11. Announce support for EML or an equivalent open standard format as a procurement 

9 requirement for new voting systems — specifically, as the format for election definitions, 

10 results, outcomes, and any reports necessary to support the risk-limiting audit procedure 

11 in use. 

12 12. Announce an acquisition preference for voting systems that allow individual voters to 

13 verify their cast votes after the election and independently check the vote tally, without 

14 enabling voters to sell their votes or fraudulently claim that their votes were miscounted. 

15 



Page 20 of 57 



1 2.2 Balloting Systems & Services 

2 2.2.1 Introduction 

3 This section addresses selected issues and opportunities for balloting systems and services, 

4 which the VSTF believes are the most important to consider in any next-generation elections 

5 administration and voting systems platform. Where possible, this section makes tactical 

6 recommendations that can be applied to the current system(s) in place. The majority of this 

7 material, however, focuses on recommendations to guide the defining of requirements and 

8 specifications for any future voting system acquisition to enhance, extend, or replace what 

9 San Francisco currently has deployed. 

10 "Balloting systems and services" as used in this Report refers to those technologies employed 

1 1 for the following uses of secret ballots in a public election: (1) producing ballots prior to an 

12 election or on-demand during an election; (2) delivering a ballot to a voter, either in person or 

13 remotely for absentee voters; (3) marking a ballot, whether manually marking a paper ballot, 

14 digitally marking an electronic ballot, or using digital means to indicate ballot choices that are 

15 then represented on a printed ballot; (4) presenting a ballot to be counted, whether remotely or 

16 in-person, or presented physically or digitally; and (5) the actual counting of ballots. 

17 The "secret ballot" is used here in recognition that it has been traditionally a term of art within 

18 the elections administration community. However, it is important to balance the access to and 

19 verifiability of election records (including ballots) with the need to maintain voter privacy. The 

20 principal quality of the U.S. election system that guarantees this privacy is perhaps best 

21 described more constructively as "ballot anonymity" rather than "ballot secrecy." Ballot 

22 anonymity means that voters cannot provably be associated with their vote data. It does not mean 

23 that ballots should be kept secret after casting. Incidental recognition of a ballot image or a vote 

24 pattern by the voter is probably unavoidable without closing off access in an irrevocable manner. 

25 Even so, there are means available to reduce the likelihood that personal identities will be 

26 associated with otherwise anonymous records. 

27 Ballot anonymity should be recognized as an essential part of any existing or future voting 

28 system. There is concern that coercion and vote buying/selling might occur when specific ballots 

29 can be linked to specific voters. (See recent cases of vote buying and selling in Kentucky for an 

36 

30 example.) This danger, however, should not be over interpreted in a way that leads to the loss 

31 of necessary transparency in the counting of ballots or the verification of counts. At the same 

32 time, it is important to consider dangers inherent in selective access to election records (e.g. for 

33 officials only). 

34 



See http://www.kentucky.eom/2010/l 1/03/1 506063/voting-going-smoothly-across-kentucky.html. 
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1 2.2.2 Concepts and Definitions 

2 • Direct-Record Election device (DRE): a computer-based device that presents a ballot as 

3 a series of ballot items; accepts voter selection(s) for each ballot item; provides 

4 navigation, help, confirmation and other UI functions; records an electronic ballot that 

5 comprises all of a voter's ballot item selections. Some DREs include a voter-verified 

6 paper audit trail (VVPAT). 

7 • Remote Digital Voting: is a voting method in which marked ballot data is digitally 

8 transmitted from voter to election officials, either with or without a paper trail. 

9 • Uniformed and Overseas Civilian Absentee Voter Act (UOCAVA): 37 an act of the 

10 U.S. Congress that places requirements on states' conduct of elections to include 

1 1 measures to enhance access by (a) military or civilian voters not residing in the U.S. or 

12 (b) military voters on service away from their locality of voter registration. 

13 • Vote By Mail (VBM): a voting method by which a blank ballot and voter affidavit are 

14 sent via postal service to an absentee voter, who is expected to complete both documents 

15 and return them via postal or express service, packaged in such a way that the affidavit 

16 can be viewed without viewing the marked ballot. Jurisdictions employ a wide variety of 

17 methods for packaging, for information required on affidavits, and for validation (if any) 

18 of the affidavit sometimes including a signature. 

19 • Voter-Verified Paper Audit Trail (VVPAT): a paper-based component of a DRE. 

20 Some DREs print a VVPAT for a voter to view and independently verify before casting 

21 an electronic ballot. Such VVPATs are removed from view after casts; in some 

22 jurisdictions, VVPATs are used for hand-count audits of DRE counts. 

23 A Ballot Marking Device (BMD) presents a ballot as a series of ballot items; accepts voter 

24 selection(s) for each ballot item; provides navigation, help, confirmation and other user interface 

25 functions; records the voter's selections by printing a paper ballot that the voter can cast in the 

26 same manner as paper ballots that were marked by hand. Some BMDs print only selection marks 

27 (e.g. bubbles) on pre-printed ballots; other BMDs print a complete ballot on a blank sheet(s) of 

28 paper. 

29 The Central Count Optical Scan device (CCOS) incorporates digital image capture and digital 

30 image processing techniques to acquire an image of each sheet of a deck of paper ballots, 

3 1 identify voter marks on the ballot, and interpret each mark as a choice for a particular contest's 

32 candidate or choice. The votes from each scanned and counted ballot are tallied to produce vote 

33 totals from the set of ballots scanned during a single run of the device. Some CCOS devices 

34 retain ballot images and/or individual records of each counted ballot. Some CCOS devices reject 

35 ballots with ambiguous marks, while others provide a user interface for election officials to 

36 interpret the voter's intent and indicate how an ambiguous mark should be realized and recorded 

37 as a vote or non-vote. 



38 



Uniformed and Overseas Civilian Absentee Voter Act; 
http://web.archive.Org/web/20080126231627/http://www.fvap.gov/laws/uocavalaw.html . 
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1 The Precinct Count Optical Scan device (PCOS) is similar to a CCOS device, except that a 

2 PCOS device works on individual paper ballots rather than a deck of ballots, and the intervention 

3 for ambiguous ballots is to offer the voter (presumed to be present while the ballot is scanned 

4 and counted) options to re -try with a new or updated ballot, or to proceed with counting despite 

5 ambiguous marks that might result in some of the voter's votes not being counted. 

6 An agency of the U.S. Federal government, Federal Election Assistance Commission (EAC) 

38 

7 was created by the Help America Vote Act (HAVA) of 2002 with the task of assisting state and 

8 local election administration organizations in improving their capability to conduct U.S. 

9 government elections. The EAC primarily funds state and local election administration 

10 organizations, but it also awards research contracts for investigation of election-related matters. 

1 1 The EAC has funded the replacement of voting systems for much of the country, notably 

12 including voting systems that meet HAVA mandates for accessibility. 

13 The Federal Write-In Absentee Ballot (FWAB) is a paper form that UOCAVA voters may use 

14 (a) to fill out an absentee voter affidavit, and (b) to write a list of ballot items and the voter's 

15 choice for that ballot item. Use of an FWAB requires that the voter have independent knowledge 

16 of the items on the ballot that the voter is entitled to vote. If a voter makes errors in following the 

17 instructions for the affidavit, including the State-specific requirements, then local election 

18 officials may choose not to count the voter's ballot. 

19 DRE Double Commit refers to a DRE function that creates a risk for disenfranchisement. With 

20 some DREs, when a voter casts a ballot, the voter is prompted to confirm that they are finished 

21 voting, and then prompted a second time to commit and cast the electronic ballot. The 

22 disenfranchisement risk arises in practice because voters sometimes leave the polling place after 

23 the first confirmation, but without responding to the prompt for the second confirmation. At that 

24 point, the DRE will eventually time-out the voter session and not cast or count the ballot; also, 

25 until that time, poll workers have the opportunity to cast the ballot, either as is, or with 

26 modifications to the voter's selections. 

27 2.2.3 Findings 

28 2.23.1 Ballot Accessibility and Availability 

29 The current state of ballot accessibility and availability issues apply distinctly to three categories 

30 of voters: 

31 • Local In-person Voter: Voter and ballot information is provided by postal distribution 

32 and Web publication of personalized sample ballots that are close facsimiles of the actual 

33 paper ballots. 

34 



Federal Election Commission, "Help America Vote Act of 2002"; http://www.fec.gov/hava/law_ext.txt . 
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1 • Uniformed and Overseas Voter: Voter and ballot information as well as the official 

2 vote -by-mail blank ballot with an associated attestation document are made available by 

3 postal distribution at least and digital means at best to be compliant with Federal MOVE 

4 Act regulations. 39 

5 • Other Absentee Voters: Voter and ballot information as well as the official vote-by- 

6 mail blank ballot with an associated attestation document are provided by postal 

7 distribution. Applications materials for absentee voter status are available by Web 

8 download for preparation and return via postal service or in-person delivery. Special 

9 needs voters are able to obtain assistance in ballot marking and casting only if they are 

10 physically able to make it to a public polling place. These voters' only option is to rely on 

1 1 paper vote-by-mail ballot if they are able. 

12 Finding 1: Voter and ballot accessibility and availability are tailored to different types of 

13 voters' needs, but there are still areas that could be improved when voters need assistance 

14 completing a ballot (e.g. special-needs voters using vote -by-mail ballots). 

15 2.2.3.1.1 DRE Devices for Special-Needs Voter Ballot Casting 

16 In addition to those issues discussed above, there are issues pertaining to accessibility and 

17 usability of the ballot itself. 40 These devices do not produce a durable paper ballot of record 

18 equivalent to ballots provided to non-DRE voters. For special-needs voters utilizing DRE -based 

19 ballot casting services, there is an increased risk of loss of anonymity via a voter-sequence- 

20 correlation privacy exposure. This exposure arises in a scenario where most voters cast a paper 

21 ballot — and DREs are typically only used by special-needs voters — in small numbers. 41 Poll 

22 workers are capable of recalling or recording by name the sequence of special-needs voters 

23 and then communicating that information to staff members who have access to VVPATs or 

24 time-stamped audit logs and who can then determine how each DRE voter voted. 

25 Even when the number of DRE voters is not small, privacy exposure can occur in other ways — 

26 especially during primaries and other elections with multiple ballot-styles used in a single polling 

27 place. The privacy violation risk for the special-needs voter is one example of the importance of 

28 the principle that all voters cast a ballot in the same manner, so the ballots of some voters are not 

29 erroneously segregated, creating a risk for ballot attribution. This principle supports the use of 

30 BMDs so that voters can choose to mark manually or digitally, but the ballot is cast and counted 



California Secretary of State Debra Bowen government website (28 January 2010). 
Memo regarding "Military and Overseas Voter Empowerment Act"; 
http://www.sos.ca.gov/elections/ccrov/pdf/2010/ianuary/10042cbm.pdf . 

40 See Noel Runyan and Jim Tobias (26 July 2007), "Accessibility Review Report for California Top-to- 
Bottom Voting Systems Review"; http://www.sos.ca.gov/voting-systems/oversight/ttbr/accessibility- 
review-report-california-ttb-absolute-fmal-versionl6.pdf . 

41 Note: For example, if a few hundred people in a polling place vote on paper and PCOS, and a handful of 
special-needs voters use a DRE, then poll workers know that the handful of special-needs voters cast that 
handful of votes. Or, if one person is the only one to vote for a particular party, then poll workers would 
know who cast that single ballot. However, when BMDs are used (as discussed elsewhere in this Section), 
voters can choose to mark manually or digitally, but the ballot is cast and counted in the same manner 
regardless. 
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1 the same manner regardless. The advantage of a BMD over a DRE is that a paper ballot is 

2 produced for counting, audit, and verification purposes. 

3 Finding 2: DRE devices incur a risk of loss of voter anonymity. 

4 Finding 3: A BMD could provide a record for each ballot. 

5 2.2.3.1.2 Physical Durability of VVPAT Ballots 

6 DRE -voters are disadvantaged in audits or recounts due to the less durable nature of a WPAT 

7 ballot compared to standard paper ballots. Moreover, VVPAT rolls of paper are difficult to count 

8 in the case of manual recounts and full recounts. 42 

9 Three usability limitations should be addressed: 

10 1 . The need exists to verify instruction text meets EAC guidelines 43 for plain-language and 

1 1 moderate-level literacy accessibility. 

12 2. The need exists to verify visual aids exist in instruction text. 

13 3. There is likelihood that ballot layout does not meet guidelines of EAC-funded AIGA 44 

14 best practices in ballot design. 

15 Finding 4: VVPAT ballots have questionable limitations in design and physical 

16 makeup. 

17 2.2.3.1.3 Other Supportive Rationale 

18 The VSTF also located three areas where improvements in the system could be beneficial to 

19 voters. 

20 The State of California historically asserted compliance to the MOVE Act, which is a 45-day 

2 1 advance availability provision by postal distribution means of absentee voter materials for 

22 UOCAVA voters. Nevertheless, exploring opportunities to make these materials more readily 

23 available by digital means pursuant to the MOVE Act could better serve U.S. overseas and 

24 military voters. 

25 Special-needs voters tend to be disenfranchised should their individual situation prevent their 

26 ability to travel to a polling place to cast their ballot. Mobile accessible balloting services could 

27 provide these voters with a more accessible means to vote. 

28 As an equal protection principle, consistent enfranchisement depends on consistent ballot format 

29 and ballot counting procedures. This principle is not currently met in practice because some 

30 voters have their votes counted from paper ballots while other voters have their votes counted 

3 1 relying on VVPAT devices. Therefore, aspiring to a single ballot design, layout, and presentation 

32 for the ballot of record can achieve the long-term recommendation. 



For example, see the VVPAT section on this page: http://www.countedascast.com/issues/audits.php . 
43 EAC guidelines, last accessed on June 23, 201 1; 

http://www.eac.gov/testing_and_certification/voluntary_voting_system_guidelines.aspx . 
44 See generally: http ://www.aiga. org/ content.cfm/ election-proj get . 
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2 



Finding 5: U.S. overseas and military voters could be better served with a digital means 
of receiving election information and a blank ballot. 



4 

5 



3 



Finding 6: Special-needs voters could benefit from mobile accessible balloting services. 

Finding 7: Consistent ballot format and counting procedures can help to maintain voter 
enfranchisement. 



6 2.2.3.2 Ballot Marking and Casting 

7 The current state of ballot marking and casting can be divided into three methods of voting: 

8 in-person; remote; and use of the Federal Write-in Absentee Ballot (FWAB). 45 

9 In-person voting involves the casting and counting of ballots in person in polling places using 

10 two methods: (a) precinct optical scan of hand marked ballots, and (b) use of DRE devices for 

1 1 digital casting and counting. In addition to the these methods, some voters are required to vote 

12 provisionally by casting a hand-marked paper ballot that is not counted in the polling place but 

13 may be counted centrally, if approved by election officials. San Francisco also employs central 

14 count optical scan for vote-by-mail ballots and provisional ballots that have been approved by 

15 elections officials. 

16 There is significant controversy regarding the security risks of remote voting. It is well settled 

17 that marking ballots in an uncontrolled environment is vulnerable to fraud and coercion — mostly 

18 during transportation (of ballot from voter to election officials) wherein marked ballots are 

19 subject to risks that are not present in ballots marked in a controlled environment. 

20 Similarly, in discussing remote digital voting, it is well settled that all forms of remote digital 

21 voting also share these vulnerabilities, although there is significant controversy regarding scope 

22 and scale of the security risks of each form of remote digital voting, as compared with non- 
23 digital remote voting. Among the risks specific to remote digital voting are insider technical 

24 threat and Internet accessibility of remote digital voting systems to adversaries. Insider technical 

25 threat is the expansion of the scope of trusted insiders to include IT operations staff charged with 

26 managing remote digital voting systems, as well as anyone who is able to obtain IT operations 

27 privileges. Internet accessibility is a necessary consequence of using public networks for 

28 communication between remote voters and local election officials; anyone anywhere with 

29 Internet access has the ability to target remote digital voting systems in order to carry out the 

30 same type of Internet -based attacks that have succeeded against several organizations with 

3 1 security expertise that far exceeds that of any voting system vendor or election jurisdiction — 

32 including Google, 46 Adobe, RSA Security, 47 and dozens of other large corporations. With the 

33 digital-specific risks, both ballot anonymity and ballot integrity are at risk in many ways that are 

34 not applicable to ballots marked in a controlled environment with controlled transportation to 

35 election offices facilities. 



Federal Voting Assistance program website; http://www.fvap.gov/FWAB/fwab-ca.html . 
46 Zetter, Kim (14 January 2010), "Google Hack Attack Was Ultra Sophisticated, New Details Show," 
Wired; http://www.wired.com/threatlevel/2010/01/operation-aurora . 

47 Zetter, Kim (7 June 201 1), "RSA Agrees to Replace Security Tokens After Admitting Compromise," 
Wired; http://www.wired.com/threatlevel/201 1/06/rsa-replaces-securid-tokens/ . 
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1 An interesting example was the Okaloosa Distance Balloting Pilot, which used a combination 

2 of early-voting center operations, kiosk-style Internet voting in controlled environment, and 

3 paper ballot-like voter-verified paper records used for auditing the Internet voting tallies. More 

4 recent proposals for digital-enabled kiosk voting have included methods that do not rely on 

5 Internet voting techniques. In any event, the concepts of controlled environment and a verifiable 

6 paper trail and audit trails have emerged as the top issues wherein any remote voting solution is 

7 contemplated. 49 ' 50 

8 These issues were highlighted in the Okaloosa report that noted the system is vulnerable to attack 

9 by trusted insiders (such as election officials behaving maliciously). Defending against such 

10 attacks can be challenging in any voting system. In Scytl's system, Voter Choice Records are 

1 1 pivotal to this defense. Manual counts of the Voter Choice Records, as well as procedural 

12 controls on insider access to the system before and during an election, are the only way we have 

13 identified to secure the system against insider threats. We also note that an EAC report reported 

14 irregularities in the post-election audit of the Voter Choice Records. 51 There are a number of 

15 open issues to be resolved, including but not limited to scalability, transparency, and independent 

16 testing. 

17 Federal Write-In Absentee Ballot (FWAB) is a method that is approved by a process similar to 

1 8 vote-by-mail process, but it requires manual intervention for counting purposes. 

19 Finding 8: Although all voting methods must be carefully monitored to prevent 

20 malicious or negligent events, the use of remote digital voting — especially the digital 

21 return of voted electronic ballots with no audited paper ballots — is far too insecure in 

22 public elections application for the foreseeable future. 

23 2.2.3.2.1 Other Supportive Rationale 

24 There are several ways to ensure all voters have equal protection and enfranchisement. A single 

25 kind of ballot and a single method of counting can be supported along with support for 

26 accessibility. As mentioned earlier in this section, a BMD ensures two principles: (a) special- 

27 needs voters obtain automated assistance in ballot marking; and (b) all voters have a paper ballot 

28 that is consistently counted in the same manner. Ballot image retention can also be used for 

29 improved audit and verification. Moreover, CCOS logging capability can provide improved 

30 accountability, audit, and verification. 

31 Finding 9: The use a single type of ballot, BMDs, image retention, and CCOS logging 

32 can equally protect and enfranchise all voters. 



Okaloosa Distance Balloting Pilot, last accessed on June 23, 201 1; http://election.dos.state.fl.us/voting- 
systems/pdf/ODBPplanJunel 9.pdf . 

49 See http://www.operationbravo.org/documents/NASS%20VP%20Briefmg.pdf. 
50 See http://election.dos.state.fl.us/voting-systems/pdf/ODBPplanJune 19.pdf. 
51 See http://www.eac.gov/assets/l/AssetManager/Martha%20Mahoney%20- 
%20Comment%20on%20Pilot%20Proiect%20Testing%20and%20Certification.pdf . 
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1 2.2.4 Recommendations 

2 2.2.4.1 Ballot Accessibility and Availability 

3 2.2.4.1.1 Near-Term Recommendations 

4 To improve ballot accessibility and availability in the near future, the VSTF offers the following 

5 recommendations: 

6 1 . Support provisions of the Federal MOVE Act regulations for digital blank ballot 

7 distribution. 

8 2. For special-needs San Francisco-based voters who are physically unable to cast their 

9 ballot in a polling place, experiment with mobile accessible ballot marking and printing 

10 services. 

11 3. Promote the opportunity for San Francisco voters to access voting information online, 

12 including sample ballots. 

13 4. Adopt a stronger privacy-enhancing procedure that requires a larger minimum number of 

52 

14 voters using the DRE machines in order to reduce the risk of ballot attribution. Enhance 

15 poll worker training to stress this procedure and the need to comply with it. Measure 

16 compliance, and publish compliance findings, based on polling-place records of number 

17 of checked in voters and number of DRE voters. 

18 5. Create, train, and enforce a requirement that the accessible voting system be set up and 

19 working (according to specific criteria communicated in poll worker training) before the 

20 polling place is opened for general voting at the start of the Election Day. 

21 2.2.4.1.2 Longer-Term Recommendations 

22 We recommend these long-term actions: 

53 

23 6. Extend the intent of the California Election Code Section 15360 by requiring the ballot 

24 of record be specifically a paper record of uniform style, layout, and presentation 

25 consistent with its hand-marked counterpart instead of a paper artifact fulfilled by 

26 WPAT devices. 

27 7. Use paper ballot layout practices and/or tools that follow the EAC guidelines on visual 

28 design and plain language, and deliver these benefits to all voters. 

29 8. Rather than providing polling-place disabled access via DREs, instead provide access via 

30 the combination of (a) ballot-marking devices for enhanced access to the ballot, and (b) 

3 1 use of the same precinct-count casting method used by voters without special needs. In 

32 addition to removing a ballot anonymity threat of DREs, this approach would have 

33 additional benefits: lacking the so-called "double-commit issue" of DREs; providing for a 

34 digital count for audit purposes; and adhering to the EAC guidelines on visual design and 

35 plain language. 



See Footnote 41. 

^California Legislative Counsel government website, "Election Code Section 15360"; 
http://www.leginfo.ca.gov/cgi-bin/displaycode?section=elec&group=l 500 l-16000&file=l 5360 . 

Page 28 of 57 



1 2.2.4.2 Ballot Marking and Casting 



2 2.2.4.2.1 Longer-Term Recommendations 

3 9. The official "ballot of record" should be a paper artifact in uniform design, layout, and 

4 presentation consistent with its hand-marked counterpart, in order to enable a consistent 

5 method of counting, audit, and verification as well as to ensure a consistent method of 

6 ballot anonymity. 

7 10. Enhanced access to ballots should be provided by non-tabulating ballot marking devices 

8 rather than tabulating DREs. 

9 11. All in-person voters should have the options of either marking paper ballots by hand, or 

10 via the use of a ballot-marking device. 54 

1 1 12. Encourage voters who use BMDs to review their printed ballots before casting. 

12 13. All optical scanning devices should retain a good-resolution scanned image of each 

13 ballot, together with a complete cast-vote record for auditing support. 

14 14. CCOS devices should provide a user interface for election officials to interpret 

15 ambiguous ballots as needed — with full logging of every interpretation — and that said 

16 logs should be publicly available. 

17 15. If not done so already, provide data to track cases of UOCAVA voters receiving absentee 

1 8 voting materials, but not having a ballot arrive in time to be counted. 

19 



Note: In California, the voters do have the choice of using paper ballots or DREs with VVPATS. 
However, as a policy matter, the use of DREs is discouraged since all votes cast on a DRE with VVPAT 
must be counted by hand. 
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1 2.3 Security 

2 2.3.1 Introduction 

3 Elections security is vital to protect each voter's rights and assure the integrity of election data. 

4 Security throughout the election cycle — including use of the voting systems — must be 

5 implemented with procedures. Equally important is the security of the voting system's design, 

6 engineering, and manufacture — all elements are fundamental in garnering the trust voters must 

7 have in the system they are using to cast and count their ballots. Essentially, each voter relies on 

8 the soundness of the security of the voting procedures and the use and design of the system to 

9 ensure his or her vote is counted. If not, the integrity of elections and the jurisdictions that 

10 manage them can be compromised. 

1 1 Steps must be taken to build and maintain the voter's trust that (a) the digital chain of custody 

12 has not been broken and (b) no event has occurred that might affect the integrity of the election 

13 data. Unfortunately, for both physical ballots and the voting system, there are opportunities for 

14 fraud or error. The VSTF has scrutinized the issue of security and has determined 

15 recommendations that may lead to safer and more secure elections. 

16 2.3.2 Concepts and Definitions 

17 When considering voting system security, the vulnerabilities throughout its use in the election 

18 cycle must be examined. The following are major parts of the end-to-end election process that 

19 must be considered in system and procedural security: 

20 • Cryptography: protecting data from theft or alteration by transforming it (encrypting it) 

21 into an unreadable format, called cipher text that requires a secret key to decipher (or 

22 decrypt) the data back into plain text 

23 • Ballot Definition: the description of the ballot for ballot cards and for the digital vote 

24 records 

25 • Logic and Accuracy (L&A) Testing: pre-election testing of voting system elements and 

26 devices to assure that cast votes will be properly recorded in the voting system 

27 • Vote Capture: the point at which the voter's intent becomes a digital record, which will 

28 ultimately be aggregated with other votes to determine the election result 

29 • Vote Transmission: the movement of electronic data to an electronic/digital data store so 

30 that all votes for San Francisco can be read by a computer that tabulates the election 

3 1 results 

32 • Vote Tabulation: the tallying of ballots to determine the result for each election contest 

33 For paper ballots, the precinct- or central-ballot optical scanner device (e.g. Sequoia Eagle and 

34 400C, respectively) translates the marked, paper ballot to a digital record of the vote. When a 

35 direct recording electronic device (e.g. Sequoia Edge DRE) is used, the digital vote record is 

36 created by touching the devices screen to cast a vote that also produces the voter verifiable paper 

37 audit trail (VVPAT). In advance of use for an election, all of these machines undergo a L&A test 

38 and recalibrated or repaired as needed to assure they are fit for use in the election. 
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1 Data can be sneaker netted (downloading data to a device that is physically transported to 

2 another location and connected to another device for data upload) or may be transmitted 

3 electronically over a network. In San Francisco, the data recorded by the precinct optical scanner 

4 and the precinct DRE (Sequoia Eagle and Edge respectively) is saved to a removable memory 

5 pack that is transported from the precinct to the election center for upload to the central election 

6 data store. San Francisco processes vote-by-mail ballots and validated provisional ballots at the 

7 election center with a large, fast optical scanning machine (Sequoia 400C) that transmits data to 

8 the central data store over a private computer/data network of CCSF. 

9 For contests that are determined by a plurality, this is a matter of summing of the votes to 

10 determine passage of a measure or winner of a race. For Ranked-Choice Voting (RCV) — when 

1 1 there is no one candidate who received 50% +1 vote as a first choice — computer algorithms are 

12 then used to eliminate candidates and redistribute votes where needed for the voter's second- or 

13 third-choice candidate. 55 

14 2.3.3 Findings 

15 Without proper system security, handling of physical ballots can be open to fraud and error; 

16 however, malicious manipulation or negligent management of an electronic version of ballot data 

17 can be executed in greater volume, be more precise in intended impact, and be harder to detect. 

18 Thus, the level of security in voting systems is essential to assuring an accurate, correct election 

19 outcome and in garnering public trust in the election outcome. Effective procedural measures 

20 must be implemented throughout the election process to bolster security and to detect issues. A 

21 voting system that is designed with security — which is integrated into all of its elements 

22 (hardware, software, firmware, data, and network) — that supports effective security procedures 

23 will improve voter confidence in the system and election outcome. A system that is designed in 

24 concert with effective security procedure can reduce the cost of manual procedures required for 

25 security assurance of a system that has poor system security. 

26 Generally, the focus of voting system security is on preventing malicious or negligent events that 

27 cause corrupt or inaccurate voting data or otherwise disrupt the ability for a jurisdiction to obtain 

28 an accurate election result from its election system. Unfortunately, the jurisdiction conducting an 

29 election cannot rely solely on preventive security measures because a completely invulnerable 

30 system is impossible to construct. Thus, the review and audit of the election and system 

3 1 information are essential procedures in (a) providing the assurance that security measures were 

32 successful or (b) determining that events had transpired that somehow compromised the system. 

33 Only with this step is the security regimen complete. 

34 Finding 1: Security must be considered in every feature of a voting system to ensure 

35 voter confidence. 

36 Finding 2: A voting system that is designed to be highly secure and designed in concert 

37 with security procedures can reduce the cost of security assurance. 



See "Section 2: Election Records and Post-Election Audit Procedures" for a more detailed definition of 
RCVs. 
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1 2.3.3.1 San Francisco's Current Voting System: Existing Security Issues and 

2 Mitigation 

3 23.3.1.1 San Francisco's Procurement Action and Voting System Security Concerns 

4 In 2002, the Federal government mandated a modernization of voting systems with the 

5 enactment of the Help America Vote Act (HAVA); funds were allocated for implementation of 

6 this mandate. 56 HAVA was timely law for San Francisco, which needed to replace an aging 

7 voting system for which its maintenance contract was about to expire. As detailed in the 

8 Introduction of this report, San Francisco issued a Request for Proposal (RFP) 57 for the 

9 procurement of a new voting system in May 2005. The RFP consisted of an Introduction and 

10 15 appendices that totaled 197 pages. Appendix E "Design, Fabrication and Performance 

1 1 Requirements" (25 pages) contains all requirements, including security. This was not due to 

12 disinterest on the part of San Francisco or its Department, of Elections (DOE) on the importance 

13 of security, but it does reflect the reliance on the vendor and other agencies to detect and correct 

14 security flaws. 

1 5 The systems that could be implemented to satisfy HAVA requirements and were certified for 

16 both Federal and California elections were few. Only two vendors responded to the San 

17 Francisco's RFP: Sequoia Voting Systems and ES&S. Public objections to the vendors — which 

1 8 were primarily rooted in transparency and security concerns — stalled execution of the contract 

19 for 15 months. However, because no other certified voting systems were available and no viable 

20 alternatives were emerging, San Francisco proceeded with the Sequoia procurement. From the 

21 standpoint of DOE, this was the prudent course of action: (a) it would bring the DOE into 

22 compliance with Federal law, and (b) it would serve its operational needs. Any additional 

23 consideration of security was unnecessary and superfluous to fulfillment of its legal obligations 

24 under HAVA and support to its operational mission. 

25 23.3.1.2 Top-to-Bottom Review 

26 In January 2007, Debra Bowen was sworn in as the California Secretary of State (CA SoS); on 

27 that day, she reiterated her campaign promise to assure transparency in the voting systems used 

28 in California. 58 She created a project known at the "Top-to-Bottom Review" (TTBR) 59 of the 

29 voting systems certified for use in California. The TTBR consisted of a review of software, 

30 accessibility, documentation, and a security evaluation. It eventually evidenced many security 

3 1 issues within California's voting systems, including the Sequoia system procured by San 

32 Francisco. Security issues were found with all systems that were tested, but we here focus on the 

33 Sequoia System used in San Francisco to provide some relevant and insightful specifics on 



56 Federal Election Commission, "Help America Vote Act of 2002," http://www.fec.gov/hava/law_ext.txt . 
57 City and County of San Francisco government website, "RFP," (VSTF page, Appendix E) 
http://www.sfgov2.org/index.aspx?page=1869. 

58 California Secretary of State Debra Bowen government website, "Secretary of State Debra Bowen, 
Inaugural Speech" (Monday, January 8, 2007); 
http://www.sos.ca.gov/bowen event/inaugural speech.pdf. 

59 California Secretary of State Debra Bowen government website, "Top-to-Bottom Review"; 
http://www.sos.ca.gov/voting-systems/oversight/top-to-bottom-review.htm . 
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1 security flaws of existing voting systems. The "Source Code Review of the Sequoia Voting 

2 System" of the TTBR's Executive Summary 60 pinpointed serious security issues concerning data 

3 integrity, cryptography, access control, and software engineering. 

4 2.3.3.1.2.1 Data Integrity Flaws 

5 The review discussed how the Sequoia system "lacked effective safeguards against corrupted or 

6 malicious data" that was injected into removable media. This was a particular issue with the 

7 devices used by polls workers and other temporary staff with limited authority. 

8 Finding 3: The Sequoia voting system's lacked effective safeguards against corrupted or 

9 malicious data into removable data recording media. 

10 2.3.3.1.2.2 Cryptography Problems 

1 1 The review also stated that many of the security features of the Sequoia system — particularly 

12 "those that protect the integrity of the precinct results" — employed cryptography. Every case that 

13 the TTBR examined proved how simple it was to circumvent the cryptography. As the review 

14 explained, many cryptography functions are not implemented correctly, are based on weak and 

15 flawed algorithms, or are used in an ineffective or insecure manner. Because of these issues, 

16 "virtually all cryptographic key material is permanently hardcoded into the system" and identical 

17 in all of the hardware that was shipped off to other jurisdictions. In short, a person who is able to 

18 hack into a similar hardware — within or outside of California — can then extract and obtain the 

19 secret cryptographic key that were initially created to protect elections throughout every 

20 California county that employs that system. 

21 Finding 4: Sequoia's cryptography was poorly implemented, hard coded into the system, 

22 and identical in all of the Sequoia systems used throughout California. 

23 2.3.3.1.2.3 Access Control and Security Mechanisms Issues 

24 The TTBR also discovered issues with access control and other computer security 

25 mechanisms that were easily circumvented — despite being designed to protect against 

26 "unauthorized use of central vote counting computers and polling place equipment." The 

27 WinEDS back-up system was designed to be used for ballot preparation, voting machine 

28 configuration, absentee ballot processing, and post-election vote counting. However, its 

29 security features and audit logs were found to be ineffective against "inside attackers" 

30 who may try to gain access to the WinEDS computers or the network to which these 

3 1 computers are attached. 

32 Finding 5: The security features and audit logs of the WinEDS back-up system 

33 could have been easily comprised by insiders. 



Blaze, Matt, Arel Cordero, Sophie Engle, Chris Karlof, Naveen Sastry, Micah Sherr, Till Stegers, and 
Ka-Ping Yee (2007). "Source Code Review of the Sequoia Voting System," p. 2 (report, University of 
California, Berkeley under contract to the California Secretary of State's TTBR); 
http://www.sos.ca.gov/voting-systems/oversight/ttbr/sequoia-source-public-jul26.pdf . 
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1 2.3.3.1.2.4 Software Engineering Weaknesses 

2 The software engineering of the Sequoia voting system was also found at fault by the 

3 TTBR. According to the review, the software contained numerous programming errors, 

4 many of which had the "high potential to introduce or exacerbate security weaknesses." 

5 Basically, the software did not reflect "defensive software engineering practices normally 

6 associated with high-assurance critical systems." The review also pointed out that there 

7 were many examples of poor or absent error and exception handling and that there were 

8 also many cases where the software behavior did not match its corresponding comments 

9 and documentation. Some of the problems were the root of the many of the issues the 

10 review identified, and even the problems discovered in the software that were not specific 

11 to "an obvious vulnerability identified," the number of errors reduced the review's 

12 "overall confidence in the soundness of the system as a whole." 

13 Finding 6: The software for the Sequoia voting system contained serious 

14 programming errors that reduced the overall trust in the reliability of the voting 

15 system. 

16 2.3.3.1.2.5 "Red Team" Security Testing 

17 The CA SoS's security group "acted as a 'Red Team' [penetration testers] and performed a series 

18 of security tests of both the hardware and the software," 61 concluding that — although there was 

19 not sufficient time to perform a complete evaluation of the Sequoia voting system — the number 

20 of serious security issues that were exposed was cause for concern. Essentially, a determined 

2 1 hacker could modify or invalidate the results of an election. The review impressed that several 

22 types of attacks could be launched without any knowledge of the source code. In fact, the Red 

23 Team was able to analyze the Edge's firmware binary representation and extend the firmware by 

24 using binary patching. This technique allowed them to create a "'debugging' version of the 

25 firmware, as well as several different 'malicious' versions"; again, access to the source code to 

26 implement these attacks was not necessary. 

27 Finding 7: Access to the source code was not necessary to attack the hardware and 

28 software of the voting system. 

29 Finding 8: Public concern over DOE's procurement of the Sequoia Voting Systems was 

30 not unfounded. 



31 



Computer Security Group (2007). "Security Evaluation of the Sequoia Voting System Public Report," 
Dept. of Computer Science, University of California, Santa Barbara; http://www.sos.ca.gov/voting- 
systems/oversight/ttbr/red-sequoia.pdf . 
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1 23.3.1.3 Security Mitigations Measures Required for Use of the Sequoia Voting System 

2 As a result of the TTBR's findings, on 25 October 2007 SoS Bowen issued the "Withdrawal of 

62 

3 Approval of Sequoia Voting Systems, Inc." (updated version issued 1 October 2009) — a 

4 document that also provides the requirements needed for re-approval of the system. The result 

5 was the generation of the "Optech Insight, AVC Edge 5.0, & Optech 400C California 

63 

6 Procedures," deemed the "Sequoia 4.0 Approved Use Procedures" that allowed conditional 

7 re-approval of the system and — with implementation of these procedures — the use of the system 

8 in San Francisco. 

9 San Francisco and Sequoia have implemented the mitigation plans approved by the CA SoS. San 

10 Francisco DOE maintains a Voting System Security Plan 64 that addresses policies, practices, and 

1 1 procedures for voting system security and that addresses specific requirements for continued 

12 approval for use of Sequoia by the California Secretary of State. 

13 Finding 9: The public interest would be served by raising awareness of (a) the Sequoia 

14 system vulnerabilities identified in the TTBR, (b) the mitigation measures prescribed by 

15 the CA SoS, and (c) the procedures that implement these measures in San Francisco. 

16 2.33.2 Current Voting System Security Posture 

17 Based on optical scanning of paper ballots, the fundamental security posture of San Francisco's 

18 existing counting methods consists of (1) implementing practices, policies and procedures to 

19 meet legal requirements for security, and (2) validation of machine counts by conducting partial 

20 hand counts of the vote a technology independent manner as required by California Election 

21 Law. The security practices and requirements include reducing or eliminating exposure to attack 

22 points such as connections to wireless devices or the Internet as well as using tamper-evident 

23 seals, signature checks, and other chain-of-custody procedures that increase the chances of 

24 detecting errors or tampering. 

25 "Technology independent" validation (as the phrase applies in this section of the Report) means 

26 that vote counts and election results are not produced by the sole reliance on the potentially 

27 fallible software and hardware of a voting system, but instead they are produced by a 

28 combination of the following: 

29 1 . Machine count of virtually 100 percent 65 of paper ballots 



California Secretary of State Debra Bowen government website, "Withdrawal of Approval of Sequoia 
Voting Systems, Inc., WinEDS v 3.1.012/AVC Edge/Insight/Optech 400-C DRE & Optical Scan Voting 
System And Conditional Re-approval of Use of Sequoia Voting Systems, Inc., WinEDS v 3.1.012/AVC 
Edge/Insight/Optech 400-C DRE & Optical Scan Voting System (December 31, 2009 Revision); 
http://www.sos.ca.gov/voting-svstems/vendors/sequoia/sequoia-31012-revision-1209.pdf . 
63 0ptech Insight (August 2008), "AVC Edge 5.0, & Optech 400C California Procedures," (document 
version 3.03); http://www.sos.ca.gov/voting-svstems/vendors/use-procedures/sequoia-use-procedures.pdf . 
64 City and County of San Francisco government website, "VSTF Voting System Security Plan"; 
http://www.sfgov2.org/ftp/uploadedfiles/VotingSystemsTaskForce/VotingSystemSecurityPlan.pdf. 
65 Note: If Federal Write-in Absentee Ballots (FWAB) are cast, they must be hand counted. 
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1 2. Audit of the machine counts via hand-count of a randomly selected subset of the 

2 machine-counted ballots 6 

3 The audit procedure is intended to detect discrepancies in the vote count as tabulated by the 

4 voting system versus a hand count of the ballot of record. This procedure should audit a 

5 statistically significant sample relative to the number of races and voters, and it should provide a 

6 threshold to expand the scope of the audit in the event that significant variances are detected. 

7 As already discussed with respect to security, the audit approach is a forensic method for 

8 "detection of errors" and could only discover exploitation of security vulnerabilities with 

9 secondary investigation. "Prevention of errors" by exploitation of security vulnerabilities means 

10 seeking to create a secure or trustworthy system. However, a perfectly secure system is an 

1 1 impossible goal because all software is potentially fallible. 

12 Finding 10: Basic, prudent security measures are already in practice including but not 

13 limited to (a) keeping voting systems components disconnected from public networks, 

14 and (b) checking the integrity of device firmware and/or software on voting systems 

15 components through pre-election L&A tests. 

16 2.33.3 Security for San Francisco's Future Voting Systems 

17 23.33.1 Comprehensive Voting System Security Examination Not Attempted by VSTF 

18 The VSTF did not attempt a comprehensive examination of information security as it applies to 

19 voting systems. The threefold reasoning for this became clear during the course of our research: 

20 1 . The state of the voting systems industry is bleak: only two major vendors remain, 

21 controlling some 87% of U.S. voting systems in use, with a few smaller vendors serving 

22 small pockets of opportunity. 

23 2. Voting technology experts concur that future voting systems design will require a 

24 wholesale change in the technology model as well as testing and certification methods 

25 and requirements for Federal certification in order for these systems to increase accuracy, 

26 transparency, verification, security, and, above all, the voters' trust. 

27 3. The prospective fourth version of the NIST/US EAC Voluntary Voting System 

28 Guidelines (VVSG) — which provides the most extensive set of voting system 

29 requirements, including both specifications and procedures for security — was expected to 

30 be released in 2009, but it has yet to be adopted in a final form. 

3 1 Finding 11: The VSTF found with regard to voting systems security considerations that a 

32 more focused study by more qualified security experts is necessary. 



Note: Precinct cast ballots on the Sequoia Edge Direct Recording Electronic (aka DRE) device do not 
produce a paper record that is machine read. Instead, the vote data is recorded directly to the memory 
pack that is then transported to a central location and loaded into the main tabulator along with the 
memory pack from the Sequoia Eagle Optical Scan device. The DRE does produce a paper tape record of 
the voter's selection by contest (Voter Verified Paper Audit Trail, aka VVPAT). This paper tape record 
can be used for audit purposes. 
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1 2.3.4 Recommendations 

2 2.3.4.1 Security Mitigations Measures Required for Use of the Sequoia Voting 

3 System 

4 Accordingly, this Report recommends increased transparency of and communication about San 

5 Francisco's implementation of the CA SoS's-mandated mitigations. Specifically the City should 

6 create an online resource to complement voter information resources that describe the current 

7 system, features, and functions, complete with a walk-through of the steps taken to comply with 

8 the CA SoS mandates for the current voting system. 

9 2.3.4.2 Near- to Mid-term Recommendations 

10 Beyond the immediate security concerns specific to San Francisco's current voting system, there 

1 1 are also broader concerns about information security of voting systems. The Report's near- to 

12 mid-term recommendations are that San Francisco should increase (a) public awareness and 

13 education of the security posture of computer-based vote counting, and (b) transparency of 

14 operations with regard to this posture. 

15 2.3.4.3 The Current Voting System Security Posture 

16 Many basic security posture measures are specified as TTBR mitigations, L&A testing practices, 

17 and post-election operations reviews. With that in mind, in the interim period between the 

18 current state of San Francisco's voting system and any future system to be acquired, the Report 

19 advocates the following recommendations: 

20 1 . San Francisco should further public trust by increasing communication of the basic points 

21 of the security posture and, in particular, by impressing upon the voter that — 

22 a. The creation of a perfectly secure voting system software is impossible; 

23 b. Manual audits can provide assurance of a clean and accurate election, thus minimizing 

24 the voters' focus on the correctness and integrity of software. 

25 2. San Francisco should continue to maintain the existing practices of L&A testing and 

26 TTBR mitigation. 

27 3. San Francisco should increase the operational transparency and adequacy thresholds of 

28 statistical audit practices to include the following: 

29 a. Greater information on and availability of audit results; 

30 b. Voter education about auditing and results through online resources that complement 

3 1 existing voter information services. 

32 San Francisco should consider various options for increasing the scope of audits beyond the 

33 minimum requirements of California Election Law. (See Section 2.1: Election Records and 

34 Post-Election Audit Procedures for further discussion and recommendations on this topic.) 
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1 2.3.4.4 Security for San Francisco's Future Voting Systems 

2 Accordingly, this Report's overarching recommendation with regard to voting system security is 

3 that San Francisco collaborate with or create a new, highly qualified, agile team of 4-6 computer 

4 systems scientists to develop a set of guidelines for security aspects of any future voting system 

5 to be acquired. 

6 For a procured system, these guidelines should comprise new security requirements to be 

7 incorporated into any future RFPs to be responded to by any provider of voting systems to San 

8 Francisco. Should San Francisco proceed with a decision to make a system to their requirements, 

9 these guidelines should be further developed to become requirements that are incorporated into 

10 overall systems design. 

1 1 This new "Security Guidelines Team" could be a new Task Force or simply collaboration with 

12 both academia and computer industry professionals outside of the voting systems industry on a 

13 consultative basis. It is crucial, though, that these team members have demonstrated domain 

14 expertise in elections technology and related information security matters. 67 

15 2.3.4.5 Longer-Term Recommendations 

16 Aside from assembling a team of digital security experts to develop RFP guidelines for future 

17 voting systems, this Report suggests several features that can support increased voting systems 

18 security and elections process integrity (many of which are discussed elsewhere in this Report): 

19 4. Assuring a system that allows for hand marking and machine-assisted creation of marked 

20 paper ballots of like media versus providing WPAT for ballots cast by voters with 

2 1 requirements for enhanced access 

22 5. Continuing the use of precinct-count optical scan for in-person cast ballots and central - 

23 count optical scan for absentee and provisional ballots 

24 6. Providing digital images of each counted ballot, with a cast-vote record for each that 

25 would be made available for examination 

26 7. Establishing strong protections to assure that all actions that change or update the system 

27 are known and that only approved software and hardware are implemented and used in an 

28 election with documented, approved change management procedures during an update 

29 and deployment of the system 

30 a. Voting system capabilities for strong authentication, access, and logging of system 

3 1 events and operator actions with notification and audit procedures that assure only 

32 authorized access and approved actions were taken in any part of the system 



Note: By way of example, but not limitation, three example sources of domain experts include: (a) the 
California Institute of Technology and Massachusetts Institute of Technology joint project known as the 
CalTech/MIT Voting Project (see http ://vote. caltech.edu/drupal/ ) ; (b) ACCURATE: A Center for Correct, 
Usable, Reliable, Auditable, and Transparent Elections, the organization involved with the TTBR (see 
http://accurate-voting.org/ ): and (c) The OSDV Foundation's TrustTheVote Project (see 
http://www.osdv.org and http ://www. trustthe vote . org ) . 
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1 b. Logging that should involve actions executed with respect to the system hardware and 

2 software and election data, including any change to vote records, such as resolution of 

3 under-votes, over-votes, and identification and recording of write-ins 

4 8. Utilizing election management system features and reporting system features for 

5 publication of ballot definition data and vote count data as recorded by counting devices. 

6 9. Using common data formats to facilitate publication of such data mentioned above 

7 10. Establishing features related to voting system verification loops, testing practices, and 

8 transparency of records of such practices, including (but not limited to) the following: 

9 a. Straightforward and easily repeatable measures for testing software integrity of voting 

10 system components 

11 b. Election management system features and reporting system features for recording and 

12 publishing both components of and results of L&A testing (e.g. test decks and test- 

13 count results) 

14 11. Providing a well-documented system that can be maintained and operated with 

15 commonly and widely available skill sets (versus vendor dependence that is due to 

16 proprietary elements and non-disclosure of system technology) 

17 12. Maintaining transparency throughout the system hardware, firmware and software 

18 life-cycle including system design, engineering, and manufacture as well as data formats, 

19 encryption and communications protocols, and network security requirements 

20 13. Having the ability to validate only authorized software used to execute the election in the 

21 system 



22 These capabilities and features should also be considerations of the proposed Security Guidelines 

23 Team for when those team members prepare a set of security guidelines for future RFP and 

24 competitive vendor bidding. 

25 
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1 2.4 Ranked-Choice Voting 

2 2.4.1 Introduction 

3 Ranked-Choice Voting has been the law in San Francisco since 2002. In March of that year, San 

4 Franciscans passed Proposition A, 68 amending the City Charter to make 

5 Instant-Runoff Voting (IRV) — commonly referred to as Ranked-Choice Voting (RCV) — the 

6 method of electing Mayor, Sheriff, District Attorney, City Attorney, Treasurer, Assessor- 

7 Recorder, Public Defender, and members of the Board of Supervisors. This was codified in the 

8 San Francisco Charter as Article XIII, Section 13. 102. 69 Federal and California State provisional 

9 certifications of the required changes to San Francisco's then current Elections Systems and 

10 Software (ES&S) voting system were obtained by April 2004, and this method was first used in 

1 1 November 2004 to elect seven supervisors. 70 RCV has also been implemented by more than a 

12 dozen other U.S. cities — including three in the Bay Area (Berkeley, Oakland, and San 

13 Leandro) — and some States for military and overseas voters; it is also in use in a number of other 

14 countries. 71 ' 72 

15 RCV is existing law in San Francisco. While there remains public debate about RCV as a voting 

16 method, the VSTF has assumed its use in San Francisco as a given, and thus we have limited its 

17 examination to certain aspects of RCV implementation in San Francisco that relate to voting 

1 8 systems and public understanding of the election process. 

19 2.4.2 Concepts and Definitions 

20 • Ranked-Choice Voting (RCV): allows the voter to make multiple selections in a single 

21 race in an order of preference. As currently implemented in San Francisco, voters are 

22 allowed to vote for three choices among the candidates in an RCV race in a ranked order 

23 of first choice, second choice, and third choice. 

24 • RCV Algorithm: determines the RCV winner by tabulating votes in a series of rounds. 

25 The first round of tabulation counts votes for the first choices in a race. If the top vote 

26 getting candidate also has a majority of the votes, that candidate is declared the winner. If 

27 no candidate has a majority, then the candidate with the fewest first-choice votes is 

28 eliminated from the race. A new round of vote tabulation is then performed in which each 

29 vote that had counted for an eliminated candidate is transferred to that ballot's most 

30 preferred candidate who remains in the race and the votes are again tallied to see if the 

31 top vote getting candidate has a majority of votes. If so, that candidate is declared the 



1 For complete text of Proposition A, see http://www.smartvoter.Org/2002/03/05/ca/sf/prop/A/ . 

69 For complete text of Section 13.102 — Instant Runoff Elections, see 

http://librarv.municode.com/index.aspx?clientld=14130«festateld=5«festateName=Califorma. 

70 Fair Vote: The Center for Voting and Democracy website, "San Francisco Successfully Uses Ranked 

Choice Voting for Citywide Elections, Nov. 2005"; http://www.sfrcv.com . 

71 Fair Vote The Center for Voting and Democracy, "Where Instant Runoff Is Used"; 

http://www.fairvote.org/where-instant-runoff-is-used . 

12 Wikipedia, s.v. "Instant-runoff voting, Global Use," last modified 12 June 201 1; 
http://en.wikipedia.Org/wiki/Instant-runoff_voting#Global_use . 



Page 40 of 57 



1 

2 

3 
4 
5 
6 



winner. If not, subsequent rounds of elimination and transfer are performed until a winner 
is determined. 



• Continuing and Exhausted Ballots: A ballot is "continuing" if after a round of 

elimination the ballot has valid marked choices for a candidate who remains in the race. 
A ballot is "exhausted" if after an elimination round the ballot does not have marked 
choices for a candidate who remains in the race. 



7 2.4.3 Findings 



8 2.43.1 Public Understanding of RCV 



9 In San Francisco's November 2010 elections there were contests for five County Supervisor 

10 seats that were RCV races as called for in the City Charter. In two of these contests, District 2 

1 1 and 10, the ultimate winner did not receive the most first-choice votes and thus were not the front 

12 runners in the first RCV round. This also happened in Oakland's mayoral RCV race. All of these 

13 races were competitive and close. District 10 race was perhaps most exceptional. There were 21 

14 candidates listed on the ballot. In a race where nearly 17,808 ballots were cast, there was only a 

15 181 vote spread between the top five first-choice vote getting candidates. The winning candidate 

16 had placed third when first-choice votes were tallied in the first RCV round. 

17 Since the November 2010 election cycle various sources — press accounts as well as statements 

1 8 from candidates and other organizations — have scrutinized the RCV elections process and 

19 outcome. Some are positive stating that RCV worked as expected and proved its benefits while 

20 others are critical claiming the method is undemocratic, or that RCV delivers surprise outcomes 

■7-5 

21 that voters do not understand. 

22 The San Francisco Department of Elections (DOE) is required by the City Charter Article XIII 

23 Section 13.102(g) to "conduct a voter education campaign to familiarize voters with the ranked- 

24 choice. . .method of voting." The DOE developed a training plan in advance of the first RCV 

25 election in November of 2004. 74 This has been a significant effort that produced hard copy and 

26 on-line educational materials on the RCV voting method and over 700 outreach events 

27 coordinated with 1 1 community-based organizations. 75 A study conducted by San Francisco 

28 State University after the November 2004 election concluded that 87% of voters understood 



The following is an article that is representative of various perspectives: Zusha Elinson and Gerry Shin 
(11 November 2010), "The Winning Strategy in Oakland: Concentrate on Being 2nd or 3rd Choice"; 
http://www.nytimes.eom/2010/l 1/1 2/us/politics/l 2bcvoting.html . 

74 Department of Elections Ranked Choice Voting Public Education Plan November 2, 2004 Consolidated 
General Election, 

http://politicalreform.newamerica.net/files/San%20Francisco%20Dept%20oP/o20Elections%20RCV%20 
Education%20Outreach%20Plan.pdf . 

"California Secretary of State Debra Bowen government website, "Implementation of Ranked-Choice 
Voting: The City and County of San Francisco: November 2, 2004 Municipal Election" (p. 4); 
http://www.sos.ca.gov/voting-systems/vendors/ess/rcv-fmal-report.pdf . 
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1 RCV "fairly well" or "perfectly well." Though this indicates a high degree of understanding of 

2 the process among most voters, that 13 % of voters had a lower understanding of the voting 

3 system is significant; this group tended to be (1) lower income, minority voters and 

4 (2) those who have a lower incidence of voting. In a considerably less exhaustive but more 

5 recent study by the San Francisco Chamber of Commerce "55 percent of voters say they are 

6 unsure whether or not their vote is counted if their first, second and third choice candidates are 

7 eliminated." 77 

8 The voter education materials do still exist and can be found on the DOE's website. The level of 

9 outreach and educational activity on RCV has not been at the levels they were in 2004. 

10 Finding 1: Public understanding of the RCV election process may have declined since 

1 1 the initial RCV education campaign starting in 2004. 

12 2.43.2 Reporting Preliminary Early Election Results 

13 Once votes are cast and captured in the election system as data, they are gathered into the 

14 central database to determine the election outcome. For RCV races, the software and algorithms 

15 for tabulating the election result are more complex than for elections where the outcome is 

16 determined by simply summing the votes of the choices on the ballot. This may contribute to a 

17 perception of some voters that they do not understand how RCV works. On the other hand, 

1 8 computerization of the election process has made it easier to frequently produce preliminary 

19 election results, and San Francisco DOE has set a very high standard for its frequency of 

20 publishing preliminary election results. The DOE's schedule of results reporting for the 

21 2 November 2010 election was the following: 

22 • Election Night: Preliminary results for early return, pre-processed vote-by-mail ballots 

23 and precinct-counted ballots. The first preliminary results are reported approximately 45 

24 minutes after the close of polls, and updates are reported approximately every half-hour 

25 to an hour until midnight. For RCV contests, only first-choice totals are reported. 

26 • Subsequent Days: Every day in which new votes are processed, the Department will 

27 release updated results until all ballots have been counted and the results are certified. 

28 • Preliminary Ranked-Choice Results: Release of preliminary results represent how 

29 ranked-choice voting plays out on only the votes counted to date. The first of these 

30 preliminary RCV results are released on the Friday after Election Day. 

31 



Public Research Institute website (December 2004), "An Assessment of Ranked-Choice Voting in the 
San Francisco 2004 Election: Preliminary Report" (p. 9); http://pri.sfsu.edu/reports/SFSU- 
PRI%20Ranked%20Choice%20Voting%20Preliminarv%20Report.pdf . 
77 San Francisco Chamber of Commerce (February 201 1), "201 1 City Beat Poll Results"; 
http://www.sfchamber.com/201 lCityBeatPoll/201 lpollresults.pdf . 

78 City and County of San Francisco Department of Elections (November 2010), "Schedule of Results 
Reporting for November 2, 2010 Consolidated General Election"; 
http://www.sfelections.org/ec/7rrF20 1011 . 
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1 Release of results with this frequency is a good practice for transparency. However transparency 

2 is reduced by not reporting the preliminary results with the full RCV algorithm applied, 

3 involving eliminations and transfers and the detail cast vote records on election night or with 

4 every daily update. It should be noted that in San Francisco's first RCV election in November 

5 2004, the DOE had planned to produce a "preliminary and initial RCV Algorithm report the day 

70 

6 after the election at 4:00 p.m. as well as up to three times a week until results were final." 

7 However, during the first attempt to apply "the RCV algorithm, ES&S [the voting system 

8 vendor] realized the system was not tabulating all of the processed ballots and could not produce 

80 

9 complete preliminary RCV results." ES&S attributed the issue to a software limitation which 

10 was removed and by "that Friday, ES&S isolated and removed this particular limitation on the 

8 1 

1 1 software." DOE has continued to adhere to this practice of producing the first result with the 

12 full RCV algorithm on the Friday after the election with the Sequoia voting system. 

13 As seen in November 2010 RCV contests, second and third choices have a significant effect on 

14 the outcome of an RCV contest. Not producing results with the full algorithm applied could 

15 contribute to a perception on the part of the public that they do not understand RCV. Timely 

16 disclosure of preliminary results with the full RCV algorithm applied will improve transparency 

17 and give the public a better understanding of the ultimate election result. It is also important for 

18 public monitoring of elections. Full reporting of RCV results avoids reliance on potentially 

19 misleading vote totals based only on first choices. 

20 Finally, a substantial area of findings and recommendations in this report is in Election Records 

21 and Post-Election Audit Procedures (see Section 2.1). Early release of election results with a 

22 fully run RCV algorithm is complementary to improved audit procedures. 

23 Finding 2: The Department of Elections has a good practice of frequently releasing 

24 preliminary vote counts, but it does not apply RCV algorithms at each release. And this 

25 may contribute to a perception of lack of understanding and/or transparency in the RCV 

26 election process. 

27 2.43.3 Three-Choice Limit 

82 

28 San Francisco Charter Section 13.102(b) states that: 

29 The ballot shall allow voters to rank a number of choices in order of preference equal to 

30 the total number of candidates for each office; 



31 



California Secretary of State Debra Bowen government website, "Implementation of Ranked-Choice 
Voting: The City and County of San Francisco: November 2, 2004 Municipal Election" (p. 8); 
http://www.sos.ca.gov/voting-svstems/vendors/ess/rcv-final-report.pdf . 
80 Ibid, p. 8. 
81 Ibid, p. 8. 

82 San Francisco, Calif, Charter, Article XIII, Section 13.102(b). 
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1 It is clear that the intent is to allow the voter to be afforded the opportunity to rank all candidates 

2 in a race, but the same passage goes on to state: 

3 . . .provided, however, if the voting system, vote tabulation system or similar or related 

4 equipment used by the City and County cannot feasibly accommodate choices equal to 

5 the total number of candidates running for each office, then the Director of Elections may 

6 limit the number of choices a voter may rank to no fewer than three. 

7 Thus, three choices is the minimum allowed, but this limitation should only be imposed if the 

8 voting system is not able to accommodate more choices. San Francisco has been using RCV with 

9 this minimum level of capability due to San Francisco's Sequoia voting system limitations. 

10 However, the law says the voter should be able to rank all candidates for good reasons: the 

1 1 three-choice limit imposed by the voting system limits voters full expression. In a recent court 

12 decision, the U.S. 9th Circuit Court of Appeals concluded: 

13 If aspects of the City's restricted IRV scheme. . .impose any burdens on the voters' rights 

14 to vote, they are minimal at best. Moreover, the City has advanced valid, sufficiently- 

83 

15 important interests to justify using the system. 

16 So although San Francisco's current implementation of RCV does not violate the voters' 

17 constitutional rights, "restricted IRV" does limit the voter's ability to fully express their choices 

18 in an RCV election. In the case that voters would want to express more than three choices in an 

19 RCV election and that the ballot and voting system could accommodate those choices (which is 

20 the clear intent of Section 13. 102[b] of the City Charter), the ability of voters to fully express 

2 1 those choices could materially affect the outcome of an RCV election. 

22 The November 2010 District 10 Supervisor election is an instructive example. In this race 17,808 

23 valid ballots were processed in the first RCV round. The top five candidates were separated by 

24 only 181 first-choice votes. The winner was finally determined in the 20th RCV round, winning 

25 by only 442 votes of the 8200 continuing ballots. 84 By the 20th round, 4977 ballots (28%) were 

85 

26 exhausted with less than 3 valid choices; however, 463 1 ballots (26%) were exhausted with 

27 three valid choices. It is highly conceivable that enough of the 463 1 voters whose ballots were 

28 exhausted due to the 3-choice limit would have wanted to express more than 3 choices, and — 

29 given the spread of votes throughout the RCV elimination rounds — this expression could very 

30 well have changed the outcome of the election. 

3 1 Finding 3: San Francisco's Charter states that voters should be able to rank a number of 

32 choices in order of preference equal to the total number of candidates in an RCV race but 

33 may be limited to as few as 3 choices should it be infeasible, which it currently is due to 



U.S. Courts for the 9 Circuit government website, "Court of Appeals: Dudum v. Arntz, No 10-17198, 
May 20, 201 1" (p. 32); http://www.ca9.uscourts.gov/datastore/opinions/2011/05/20/10-17198.pdf . 
84 City and County of San Francisco Department of Elections website, "Official Ranked-Choice Results 
Report, November 2, 2010: Consolidated Statewide Direct Primary Election Board of Supervisors, 
District 10"; http://sfelections.org/results/20 1011 02/data/dl O.html . 

85 City and County of San Francisco government website, "Department of Elections, San Francisco 2010 
District 10: Table of Involuntarily Exhausted Ballots" (pdf file produced by David Cary for the VSTF); 
http://www.sfgov2.org/Modules/ShowDocument.aspx?documentid=461 . 
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1 San Francisco's existing Sequoia voting system. This "restricted IRV" has been 

2 determined not to be a violation of a voter's constitutional right to vote. It does, however, 

3 impair the ability of the voter to fully express their preferences for candidates and such 

4 full expression of choices could change the outcome of RCV elections. 

5 2.4.4 Recommendations 

6 2.4.4.1 Public Understanding of RCV 

7 1 . San Francisco should reenergize its voter outreach and education efforts on RCV to better 

8 assure that voters have a good understanding of how votes are to be cast and counted. 

9 The Board of Supervisors should work with the Department of Elections to identify 

10 options and resources for renewed education efforts as soon as possible — preferably in 

1 1 advance of the November 201 1 RCV elections. DOE has good, existing materials, and 

12 social networking sites such as Facebook and You Tube could offer an opportunity for 

13 low cost, high impact outreach. The VSTF encourages the Supervisors themselves to 

14 assist in public outreach to individuals and organizations in their districts. 

15 2.4.4.2 Reporting Preliminary Early Election Results 

16 2. Continue to release preliminary results for RCV contests as frequently as they are 

17 released for non-RCV contests. Include the full RCV algorithm and the supporting detail 

18 cast vote records (aka "ballot images") as part of preliminary results. 

19 3. Implement Recommendation 2 to the extent feasible with the current system. Make this 

20 capability a requirement for any system to be acquired by San Francisco in the future. 

21 4. Assure that RCV public awareness, outreach, and education includes the information on 

22 what interim result reporting information is available as well as how to access it and use 

23 it to track and understand the RCV election process and results. 

24 2.4.4.3 Three-Choice Limit 

25 5. Explore the possibility of increasing the number of choices with the existing Sequoia 

26 voting system. Make the ability to rank more than three choices a strong preference for 

27 any future voting system to be acquired by San Francisco with a preference for a system 

28 that will allow the voter to rank all candidates in a race. 

29 
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1 2.5 Acquisition Strategies 

2 2.5.1 Introduction 

3 This section considers the voting systems "marketplace," including the state and federal 

4 regulatory/certification environment, economic considerations, and models for acquiring or 

5 developing a next generation system. It examines obstacles to innovation as well as potential 

6 partnership approaches that could break through existing barriers. It examines legal licensing 

7 options, including (1) proprietary, (2) disclosed, and (3) open source software and hardware 

8 approaches. It also puts forth software best practices that should be adopted regardless of the 

9 development strategy selected. 

10 2.5.2. Concepts and Definitions 

1 1 Public domain license: the class of license that is not limited by copyright and therefore 

12 essentially has no single owner to grant licenses. Since the work is not protected by copyright, it 

13 can be used, modified, and distributed by anyone without limitation. 

14 Open source software: a range of concepts — such as software development practices — along 

15 with licensing rules. In this document we are using the Open Source Initiative (OSI) definition 

16 of open source software; our focus is on licensing. 

17 Disclosed source license: in this document, refers to a license that gives the licensee permission 

18 to review all source code — including that of firm- ware — and the ability to share all source code 

19 with other parties. All requestors should be able to run the code for testing purposes. No one 

20 should be restricted from publishing his/her findings. The code, however, can have a proprietary 

21 license that would restrict some rights; for example, the copyright owners could require a fee to 

22 run the code in production. 

23 2.5.3 Findings 

24 2.53.1 Regulatory/Certification Environment 

25 The existing regulatory environment offers significant barriers to innovation. The certification 

26 process is shifting and cumbersome; it is extremely costly and time-consuming to bring a voting 

27 system through existing regulatory channels. 

28 



Note: For the complete OSI definition for "open source software" that the VSTF is using, see 
http://opensource.org/docs/osd. 
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1 The State of California requires both state certification and federal certification by the U.S. 

2 Elections Assistance Commission (EAC) before a direct-record election device (DRE) can be 

3 used by a jurisdiction. Testing is done by voting system test laboratories, which are labs 

4 accredited by the EAC. 88 

5 In the federal certification process, any modification to a voting system requires a re -testing of 

6 the entire system, even if the change is to an isolated part of the system. Therefore, even a small 

7 change to a voting system (such as a fix of a software defect) requires a very significant 

8 investment to achieve re-certification under the federal process. Estimates on the cost of federal 

9 certification vary; however, most estimates are in the range of $3-4 million dollars. Only six 

10 voting systems have been certified by the EAC. Congress is currently considering a proposal to 

1 1 dissolve the EAC, further contributing to lack of certainty about the future regulatory 

12 environment. 

13 The following are some of the requirements for a new voting system to be certified by the 

89 

14 Secretary of State in California: 

15 • Review of the application and documentation of the system 

16 • End-to-end functional examination and testing of the system 

17 • Volume testing under election-like conditions of the system and/or all voting devices 

1 8 with which the voter directly interacts 

19 • Security testing that includes a full source code review and penetration (red-team) testing 

20 of the system 

21 • Accessibility examination and testing of the system 

22 • Public hearing and public comment period 

23 The VSTF interviewed several Bay Area registrars as part of its research for this report, and all 

24 identified the regulatory process as a significant barrier to innovation/expansion of the 

25 marketplace. 

26 Finding 1: The existing regulatory environment creates obstacles to innovation in voting 

27 systems. 

28 2.53.2 Business and Partnership Models 

29 The voting systems industry today is highly concentrated, and there are many barriers to entry: 

30 regulatory/certification barriers (as described in the previous section); a fragmented marketplace 

3 1 with varying systems requirements; and high development costs. 

32 



87 See http://www.sos.ca.gov/voting-svstems/cert-and-approval/vsysapproval/vs-conditions.htm . 
88 See http://www.eac.gov/testing and certification/testing and certification program.aspx . 
89 California Secretary of State Debra Bowen government website, "Requirements of New Voting 
Systems"; http://www.sos.ca.gov/voting-systems/cert-and-approval/vsysapproval/vs-conditions.htm . 
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1 Currently, the State of California does not have consistent voting systems requirements across its 

2 58 counties. Each jurisdiction must initiate an independent process for establishing its needs and 

3 must negotiate independently with private vendors to acquire a voting system that satisfies those 

4 needs. While it might be preferable for the State of California (i.e. the California Secretary of 

5 State) to develop statewide voting systems requirements, no such effort is currently envisioned. 

6 Therefore, today the dominant model for implementing elections is for jurisdictions to purchase 

7 or lease proprietary voting systems from commercial vendors (see Sample Model A below). 

8 While this is currently the prevailing course of action, other models for acquiring a voting system 

9 warrant examination. Each possible approach brings a different set of economic and partnership 

10 considerations. 

11 A range of sample models includes the following: 

12 A. Purchase a Commercial Off-The-Shelf (COTS) Voting System 

13 A jurisdiction purchases a voting system (equipment and services) from a private vendor 

14 that funded its development and certification. The code is proprietary and owned by the 

15 vendor. San Francisco employs this model with Sequoia Voting Systems. 

16 B. Engineer to Order (Vendor Developed or Self -Developed) 

17 A jurisdiction establishes system requirements and either uses a Request for Proposals 

1 8 (process to select a vendor to build the voting system) or employs a full development 

19 team to build the voting system. In either case the jurisdiction owns the system. The 

20 voting system may be based on existing software components or may be built entirely 

21 from scratch. The jurisdiction funds the costs of development and certification. 

22 C. Public Partnership 

23 Jurisdictions with similar systems and regulatory requirements partner and share 

24 resources to build and maintain a voting system. The jurisdictions pool their resources to 

25 fund the costs of development and certification. 

26 D. Public/Private Partnership 

27 A jurisdiction seeks partners that may include academic institutions, non-profits, other 

28 government entities, or even private sector technology companies willing to produce 

29 non-proprietary components. Based on system requirements, the consortium develops 

30 the code and component parts. However, the code is not proprietary and the jurisdiction 

3 1 either owns the code outright or has the ability to make modifications. The potential 

32 funding for this model could vary greatly depending on the specific solution, but it could 

33 include a combination of money from jurisdictions and from donors/volunteers. 

34 There are existing non-profits that are building open source voting systems that are in various 

35 stages of readiness for elections. Two such organizations are the Open Voting Consortium 

36 (OVC) and Open Source Digital Voting Foundation (OSDV). There are also myriad systems that 

37 have been built by individuals and groups at academic institutions. Although many were built for 

38 specific research purposes and aren't made to be extended, some have the potential to be the 
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1 basis for full voting systems. The Caltech/MIT Voting Technology Project is a good source of 

2 information on existing systems. 

3 Los Angeles County is engaged in a robust effort (the Voting Systems Assessment Project or 

4 VSAP) to modernize its voting systems and is considering a variety of models/partnerships 

5 (including public/private partnerships) to acquire or develop such a system. Its goal, presented in 

6 the VSAP Incremental Plan, 91 is to have a new system in place by the end of 2015. Los Angeles 

7 is the largest and most diverse voting jurisdiction in the nation with 4.5 million registered voters. 

8 While the outcome of their VSAP is unknown and while San Francisco's needs will differ from 

9 those of Los Angeles County, their comprehensive effort has the potential to pioneer new 

10 approaches that might bring innovation to this stagnant marketplace. 

1 1 Closer to home, Alameda County shares similar systems requirements with San Francisco, 

12 notably Ranked-Choice Voting. 

13 It should be noted that, while all of the Bay Area registrars interviewed for this report were open 

14 to innovation in the voting systems marketplace, they expressed concern about the complexity of 

15 developing future voting systems using new acquisition models and getting them certified via the 

16 existing regulatory process. 

17 Finding 2: While there are barriers to moving away from the dominant model of 

1 8 purchasing a voting system from a private vendor, other acquisition models are possible 

19 and are being actively considered by other jurisdictions. 

20 2.5.3.3 Transparency, Source-Code Disclosure, Licensing, and Contingency 

21 Planning 

22 Sequoia Voting Systems developed San Francisco's current voting system using the company's 

23 proprietary system design and software development methodologies. The source code has been 

24 reviewed by some voting experts (through the California Secretary of State's 2007 Top-to- 

25 Bottom Review) and regulators, but the majority of the system is not open source and is not 

26 available for the general public to inspect. This makes it difficult for voters to establish 

27 confidence that the software is free of unknown software defects or design flaws. It is difficult to 

28 replace any aspect of the current voting system because the code is neither open source nor 

29 designed with clear modules. 

30 The ability to review source code and systems design is an essential property of a trustworthy 

3 1 voting system. By giving the public access to the source code of a voting system, there is an 

32 increased chance that a defect will be found in a voting system, whether by the election 



CalTech/MIT Voting Technology Project, last accessed on June 23, 201 1; 
http://vote.caltech.edu/drupal/. 

91 See http://www.lavote.net/General/PDFS/BOARD CORRESPONDENCE/0127201 l-054459.pdf. 
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1 administrator or a member of the public. In his paper, Hall includes ideas for contingency plans 

2 to address possible discoveries. 

3 Finding 3: Public review of source code increases the chance that defects will be 

4 identified and addressed. 



5 2.53.4 Innovation 

6 Although many jurisdictions have expressed interest in using alternative voting systems, most 

7 have not been able to go beyond researching and reporting on alternatives. Running a county- 

8 wide election is very complex, so it can be risky to try out new technologies. Several 

9 jurisdictions have tried out innovative solutions by initially testing redundant systems in limited 

10 ways in order to independently verify the accuracy of election results from the jurisdiction's 

1 1 proprietary voting systems. 

12 Innovation can be done incrementally — it does not require a new elections system to be 

13 implemented and can be achieved with innovative processes in addition to technologies. For 

14 example, San Francisco is planning to open voting locations on Saturdays in the November 201 1 

93 

15 election if the cost can be covered by private donations. This is an innovation that has the 

16 potential to increase voter turnout without requiring any new technologies. 

17 While this section has primarily discussed innovation for a jurisdiction's official results, there are 

18 several innovations for independently confirming the results of a jurisdiction's official system. 

19 One example is Takoma Park, Maryland, which used an open source system called Scantegrity in 

20 a municipal election (e.g. an election with no state or federal races). 94 Another is Humboldt 

21 County, California, that has used a project called the Humboldt County Election Transparency 

22 Project, 95 discussed in detail in Section 2.1: Election Records and Post-Election Audit. 

23 Finding 4: Innovation is possible even in conjunction with existing systems, but 

24 redundant methods of verifying the election result should be implemented whenever new 

25 innovations are tested. 



26 2.53.5 Software Best Practices 

27 There are standard software engineering best practices that have been found to create more 

28 reliable, maintainable software irrespective of the precise software development methodology 

29 used. 96 These include making sure code has ample unit-tests and is built using well-defined 

30 modules. An open source license does not ensure that code is high quality, so it is important to 



92 Hall, Joseph (2006), "Transparency Access to Source Code in Electronic Voting" (unpublished paper). 

http://josephhall.org/papers/ihall evt06.pdf. 

93 See http://www.sfgov2.org/index.aspx?page=2390 for details. 

94 See http://www.scantegrity.org/takoma/ for details on how Scantegrity was used in Takoma Park, MD. 
95 See http ://humtp. com/. 

96 For a comprehensive overview of best practices, see 

http://www.ibm.com/developerworks/websphere/library/techarticles/0306_perks/perks2.html . 
Page 50 of 57 



1 make sure that any voting system under consideration has been built using best practices that 

2 have been accepted across the software industry. 

3 2.5.4 Recommendations 

4 2.5.4.1 Regulatory/Certification Environment 

5 1 . San Francisco should advocate with the California Secretary of State and the State 

6 legislature for a new, comprehensive state certification process to replace the existing 

7 requirement for federal certification. The state should aspire to a certification process that 

8 is more agile, efficient, and cost effective to enable innovation. The new state 

9 certification process should be sound enough to ensure that any new voting system would 

10 still meet the minimum federal requirements. 

11 2.5.4.2 Business and Partnership Models 

12 2. The VSTF supports the stated intention of the San Francisco Department of Elections 

13 (DOE) to renew its contract with Sequoia Voting Systems through 2013 with the 

14 stipulation that the short-term recommendations contained in this report — particularly 

15 concerning auditing — are implemented whenever feasible. We are also open to extending 

16 this contract through 2015 if doing so would allow San Francisco to take advantage of 

17 new technologies or partnership options that would be available in the middle term as a 

18 result of Los Angeles County's VSAP project. DOE should use the intervening period to 

19 consider a broad range of possibilities regarding the business and partnership model it 

20 will pursue to acquire/develop San Francisco's next voting system, including 

21 collaborating with other jurisdictions, academic institutions, or non-profit organizations. 

22 3. To leverage its negotiating position, the DOE should consider reaching across the bay to 

23 Alameda County, which shares some similar requirements — notably Ranked-Choice 

24 Voting. 

25 4. The DOE should take current academic research, including research on risk-limiting 

26 audits and end-to-end voting, into account to ensure that this work is considered in the 

97 

27 selection of the City's next voting system. 

28 5. The DOE should closely monitor innovations in the voting systems vendor marketplace 

29 to determine if new products that meet the minimum requirements outlined in this report 

30 may be available in the required time frame. 



The Caltech/MIT Voting Project ( http://vote.caltech.edu/ ) is a good resource for current academic 
research. 
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1 2.5.4.3 Transparency, Source Code Disclosure, Licensing, and Contingency 

2 Planning 

3 6. The DOE should give strong preference to a voting system licensing structure that gives 

98 

4 San Francisco all of the rights provided by an OSI-approved license, even if the system 

5 is maintained by an external party. 

6 If an open source model is used, we recommend that the City of San Francisco work with 

7 other jurisdictions and organizations to develop and manage the code -base in order to 

8 leverage additional resources and expertise. The City of San Francisco should participate 

9 during the requirements gathering stage of development so that its unique requirements 

10 can be incorporated into the system design and implementation. 

1 1 If circumstances dictate that a solution that provides an OSI-approved license cannot be 

12 implemented by the time the contract for the City's current system expires, San Francisco 

13 should purchase voting equipment and services from a vendor who will provide a system 

14 that moves toward the following goals, irrespective of the other details of the license, so 

15 that any member of the public can perform the following tasks: 

16 • Review the source code of the entire system 

17 • Run code for testing 

18 • Distribute changes to code (i.e. documentation on defects and defect fixes can be 

19 distributed openly) 

20 7. The DOE should set up a contingency plan in case a defect is found in the source code of 

21 the voting system. The contingency plan should include a volunteer committee of experts 

22 that can rapidly address any discovered defects and recommend procedures that can 

23 address those defects. The committee of experts should include computer scientists with 

24 expertise in voting systems and security and members of the DOE with deep knowledge 

25 on the voting systems and procedures in San Francisco. 

26 8. San Francisco should be an active participant in the movement toward more open and 

27 transparent voting systems. We acknowledge the complexity of moving from the existing 

28 marketplace toward more innovative voting systems and urge San Francisco to move 

29 steadily toward the goal of transparency — even if it must do so in incremental steps. We 

30 encourage the City to be a strong advocate in the private sector marketplace for more 

3 1 transparent systems and to be open as well to new collaborative development models. 

32 2.5.4.4 Innovation 

33 9. It should be the policy of San Francisco to conduct pilot projects of alternative election 

34 technologies and procedures in municipal elections. This could initially involve a small 

35 number of precincts. These pilot projects would provide opportunities to learn how well 

36 alternative approaches work, such as using open source systems and hand counting paper 



For an alphabetical list of OSI-approved licenses, see http://www.opensource.org/licenses/alphabetical. 
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1 ballots at the polling places. All results of a pilot project should be confirmed using 

2 hand-counting. 

3 2.5.4.5 Software Best Practices 

4 10. All voting systems software should be designed and implemented using the following 

5 modern, high-quality industry methodologies: 

6 a. Peer reviews of source code should be done throughout development of the new 

7 voting system. 

8 b. All source code should include extensive unit tests. 

9 c. The system should be modular in design with open data formats for exchanging data. 

10 d. There should be well-documented code, a clear technical architecture, and a detailed 

1 1 database design. 

12 e. The system should be delivered with extensive administrative (i.e. election workers) 

13 and end-user documentation (e.g. how system will be used by voters, including voters 

14 with different accessibility requirements). 

15 
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Section 3: Appendices 
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3.1 Appendix A: San Francisco's RCV Manual Tally 
Process 

This appendix shows that the RCV manual tally process currently used in San Francisco does not 
audit the outcome of an election. Consider the following example of an RCV contest with three 
candidates (A, B, and C) and two precincts (5 ballots in Precinct 1, and 4 ballots in Precinct 2): 



Precinct 1 



Precinct 2 



first choice 
second choice 



B 



B 



B 



Manual precinct tally results: 
A has 2 first-choice votes. 
B has 2 second-choice votes. 
C has 3 first-choice votes. 
RCV: C wins in first round (3 to 2). 



Manual precinct tally results: 

A has no votes. 

B has 3 first-choice votes. 

C has 1 first-choice vote. 

RCV: B wins in first round (3 to 1 ). 



When all 9 ballots are counted together, no candidate has a majority of first-choice votes. 
Candidate A is eliminated, transferring 2 votes to Candidate B. In the second round of counting, 
Candidate B now has a majority (5 out of 9 votes) and wins the election. 



1 1 Compare this to an alternate scenario with slightly different votes cast: 



Precinct 1 



Precinct 2 



first choice 
second choice 



B 



B 



B 



B 



Manual precinct tally results: 
A has 2 first-choice votes. 
B has 2 second-choice votes. 
C has 3 first-choice votes. 
RCV: C wins in first round {3 to 2). 



Manual precinct tally results: 

A has no votes. 

B has 3 first-choice votes. 

C has 1 first-choice vote. 

RCV: B wins in first round (3 to 1 ). 



When all 9 ballots are counted together, again no candidate has a majority of first-choice votes, 
and Candidate A is eliminated. In the second round of counting, Candidate C now has a majority 
(4 out of 7 votes) and wins the election. 

Notice that in both scenarios, manual tallies within each precinct produce exactly the same 
results. The total number of first-choice and second-choice votes for each candidate is the same. 
The RCV procedure, carried out within each precinct, produces the same result. So, even a 100% 
manual tally, using the current procedure, cannot distinguish these two scenarios — yet they yield 
different winners. This demonstrates that the current manual tally procedure does not correctly 
assure the RCV election outcome. 
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1 3.2 Appendix B: Summary of Outreach 

2 In addition to welcoming public comment at its regular public meetings, the Voting Systems 

3 Task Force conducted the following research and outreach: 

4 • Roger Donaldson and Jody Sanford met with John Arntz, Director of the Department of 

5 Elections for the City and County of San Francisco. (Fall 2009) 

6 • Roger Donaldson, Jim Soper, and Ka-Ping Yee met with Lowell Finley, California 

7 Deputy Secretary of State. (9 October 2009) 

8 • VSTF member Roger Donaldson (with Jim Soper) organized and attended a 

9 demonstration of the Prime III voting system ( http://www.primevotingsvstem.com ) 

10 created by Dr. Juan Gilbert of Clemson University. This system is designed to address 

1 1 accessibility issues of concern to the disabled. (January 2010) 

12 • A public comment period on "Draft VSTF Recommendations Under Consideration" 

1 3 document was held. (February 20 1 0) 

14 • Roger Donaldson (by phone) and Ka-Ping Yee met with San Francisco DOE staff 

15 members Nataliya Kuzina and Crispin Tirso on post-election audit procedures. 

16 (23 July 20 10) ' 

17 • VSTF members organized and attended a demonstration of the Trachtenberg Election 

18 Verification System ( http://www.tevsystems.com ) at San Francisco City Hall. 

19 (15 September 2010) 

20 • Roger Donaldson, Ka-Ping Yee, and Jody Sanford met with Richard Matthews, 

21 Commissioner, San Francisco Elections Commission. (November 2010) 

22 • Beth Mazur and Jody Sanford met with Dave MacDonald, Alameda County 

23 Regis trar/Chief Information Officer. (27 January 201 1) 

24 • A public comment period on "Draft Recommendations on Voting Systems for the City 

25 and County of San Francisco" document was held. (February/March 201 1) 

26 • Beth Mazur and Jody Sanford met with Elaine Ginnold, Marin County Registrar. 

27 (1 March 2011) 

28 • Jim Soper and Jody Sanford met with Stephen Weir, Contra Costa Elections Clerk. 

29 (22 March 2011) 

30 
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Section 4: 
About the VSTF 



3 



4.1 Membership of the VSTF 



4 The Voting Systems Task Force has seven members with backgrounds in good government, 

5 computer science/software development, and accommodations serving persons with disabilities. 

6 Members serve as individuals and represent no other organization or group. The VSTF members 

7 are: 
8 

9 Jody Sanford, Chair 

10 Ka-Ping Yee, Vice-Chair 

1 1 Roger Donaldson 

12 Tim Mayer 

13 BethMazur 

14 Gregory Miller 

15 Jim Soper 
16 

17 Further information about the VSTF can be found at www.sfgov.org/vstf 



19 Roger Donaldson is currently a senior director at the Oracle Corporation where he has been 

20 employed since 1995. He began working on elections integrity and voting systems issues as a 

21 monitor with Election Protection during the 2004 Presidential election. He has served in nine San 

22 Francisco elections as a precinct inspector and field election deputy working for the San 

23 Francisco Department of Elections. He holds a Bachelor's degree in Economics and a Master's 

24 degree in Public Administration from the University of Southern California, and a Certificate in 

25 Government Contracts Administration from the University of California Los Angeles. 

26 Tim Mayer is CEO of a business services company in San Francisco. He began his voter 

27 advocacy work in 2006. He has met with numerous county registrars and various voter advocacy 

28 groups, and he attended many government and private research and information meetings. He 

29 has observed and participated in precinct and county Election Day activities. Tim has observed 

30 several demonstrations of prototypical election systems developed by both for-profit and non- 

3 1 profit organizations. In 2008 he participated in the Open Voting Consortium" demonstration of 

32 open source/paper ballot printing voting systems developed by, amongst others, VSTF member 

33 Ka-Ping Yee. 



See http://www.openvotingconsortium.org/ . 
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4.2 Biographies of VSTF Members 
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1 Beth Mazur is a technology consultant with extensive experience in open-source software and 

2 product management. She has held consulting and Product Management positions at a variety of 

3 software companies and non-profit organizations including Jaspersoft and Grameen Foundation. 

4 She has a longstanding interest in the use of technology for the improvement of the U.S. political 

5 process. Beth holds a Bachelor's of Science in Computer Science and Electrical Engineering 

6 from MIT. 

7 Gregory Miller is the CEO for the Open Source Digital Voting Foundation. He has 28 years of 

8 technical and business experience in development and eventual commercialization of the 

9 Internet. He is a trained computer scientist with graduate business education and a law degree 

10 focused on intellectual property, technology law, and public policy. He is also active in the 

1 1 American Bar Association addressing technology law and public policy issues — including 

12 Cyberlaw, Information Privacy & Security, and Internet Governance. Greg is a member of the 

13 Congressional Internet Caucus Advisory Committee and a sustaining member of the Internet 

14 Society. 

15 Jody Sanford served on the Board of the League of Women Voters of San Francisco from 2004 

16 to 2009 and was its president from 2005 to 2007. She is a communications manager with the 

17 Presidio Trust, the agency leading the transformation of the Presidio of San Francisco from a 

1 8 military post to an innovative urban national park. 

19 Jim Soper is a senior software consultant and the author of CountedAsCast.com. He is a 

20 co-chair of the Voting Systems Task Force in Alameda County, California, and a member of the 

21 California Election Protection Network's steering committee. Jim has been active in 

22 programming since the 1980s and in election integrity issues since 2005. 

23 Ka-Ping Yee received his Ph.D. in Computer Science at University of California, Berkeley, for 

24 research in usability, security, and electronic voting. He served as a source code reviewer on the 

25 California Secretary of State Debra Bowen's "Top-to-Bottom Review of Voting Systems" in 

26 2007. His dissertation, "Building Reliable Voting Machine Software," 100 examines issues of 

27 security and verifiability, and proposes simplification as the path toward high-assurance voting 

28 machines. Ka-Ping joined Google.org as a software engineer in 2008. 



29 This Report was edited by Lisa McFarren, www .mcfarrenwritingandediting .com . 



'See http://zesty.ca/voting . 
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